GitHub Actions

3 minute read

First, follow the instructions on Getting Started to sign up and create a workspace for your test suite.

Then create an API key for your workspace in the Settings area (click the cog ⚙️ icon in the sidebar) . This authentication token lets the CLI talk to your CloudBees Smart Tests workspace.

CloudBees Smart Tests record build and test results action

The CloudBees Smart Tests record build and test results action enables you to integrate CloudBees Smart Tests into your CI in simple way with less change. This action installs the CLI and runs smart-tests record build and smart-tests record tests to send data to CloudBees Smart Tests so that the test results will be analyzed in CloudBees Smart Tests to improve your developer productivity.

Example usage

name: Test

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  SMART_TESTS_TOKEN: ${{ secrets.SMART_TESTS_TOKEN }}
  SMART_TESTS_DEBUG: 1
  SMART_TESTS_REPORT_ERROR: 1

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Test
        run: <YOUR TEST COMMAND HERE>
      - name: Record
        uses: launchableinc/record-build-and-test-results-action@v1.0.0
        with:
          build_name: $GITHUB_RUN_ID
          test_runner: <YOUR TEST RUNNER HERE>
          report_path: <PATH TO TEST REPORT XML FILES>
        if: always()
▼
Test runner test_runner value Additional steps?

Ant

ant

Bazel

bazel

Behave

behave

Yes

CTest

ctest

Yes

cucumber

cucumber

Yes

Cypress

cypress

GoogleTest

googletest

Yes

Go Test

go-test

Yes

Gradle

gradle

Jest

jest

Yes

Maven

maven

minitest

minitest

Yes

NUnit Console Runner

nunit

Yes

pytest

pytest

Yes

Robot

robot

RSpec

rspec

Yes

Instructions for test runners/build tools

Android Debug Bridge (adb)

Currently, the CLI doesn’t have a record tests command for ADB. Use the command for /docs/resources/ci-tool-integrations/github-actions/#gradle[#gradle] instead.

Ant

No special steps.

Bazel

Behave

First, in order to generate reports that CloudBees Smart Tests can consume, add the --junit option to your existing behave command:

# run the tests however you normally do
behave --junit
▼

CTest

First, run your tests with ctest -T test --no-compress-output . These options ensure test results are written to the Testing directory.

cucumber

First, run cucumber with the -f junit option, like this:

bundle exec cucumber -f junit -o reports

(If you use JSON, use the CloudBees Smart Tests CLI instead.)

Cypress

GoogleTest

First, configure GoogleTest to produce JUnit compatible report files. See their documentation for how to do this. You’ll end up with a command something like this:

# run the tests however you normally do
./my-test --gtest_output=xml:./report/my-test.xml
▼

Go Test

First, in order to generate reports that CloudBees Smart Tests can consume, use go-junit-report to generate a JUnit XML file after you run tests:

# install JUnit report formatter
go get -u github.com/jstemmer/go-junit-report

# run the tests however you normally do, then produce a JUnit XML file
go test -v ./... | go-junit-report -set-exit-code > report.xml
▼

Gradle

/build//TEST-*.xml .

Jest

First, in order to generate reports that CloudBees Smart Tests can consume, use jest-junit to generate a JUnit XML file after you run tests.

# install jest-junit reporter
npm install jest-junit --save-dev
# or
yarn add --dev jest-junit
▼

You’ll need to configure jest-junit to include file paths in reports.

You can do this using environment variables:

Using environment variables
Using package.json

Recommended config:

export JEST_JUNIT_CLASSNAME="{classname}"
export JEST_JUNIT_TITLE="{title}"
export JEST_JUNIT_SUITE_NAME="{filepath}"
▼

Minimum config:

export JEST_JUNIT_SUITE_NAME="{filepath}"

Add the following lines to your package.json . The detail is the jest-junit section.

Recommended config:

"jest-junit": {
  "suiteNameTemplate": "{filepath}",
  "classNameTemplate": "{classname}",
  "titleTemplate": "{title}"
}
▼

Minimum config:

"jest-junit": {
  "suiteNameTemplate": "{filepath}"
}
▼

Then, run jest using jest-junit:

# run tests with jest-junit
jest --ci --reporters=default --reporters=jest-junit
▼

Maven

CloudBees Smart Tests supports test reports generated using Surefire , the default report plugin for Maven .

'./**/target/surefire-reports'

Note: The invocation above relies on the CLI to expand GLOBs like _ ** _.

minitest

First, use minitest-ci to output test results to a file. If you already store your test results on your CI server, it may already be installed.

NUnit Console Runner

CloudBees Smart Tests CLI accepts NUnit3 style test report XML files produced by NUnit.

pytest

First, run tests with the --junit-xml option:

pytest --junit-xml=test-results/results.xml

pytest changed its default test report format from xunit1 to xunit2 in version 6. Unfortunately, the new xunit2 format does not include file paths, which CloudBees Smart Tests needs.

Thefore, if you are using pytest 6 or newer, you must also specify junit_family=legacy as the report format. See Deprecations and Removals — pytest documentation for instructions.

Robot

output.xml

RSpec

First, use rspec_junit_formatter to output test results to a file in RSpec. If you already have a CI server storing your test results it may already be installed:

bundle exec rspec --format RspecJunitFormatter --out report/rspec.xml

Introduction