CI tool integrations

Although CloudBees Smart Tests works with any CI tool, we are working on direct integrations to make onboarding easier. Follow the instructions on Getting started to do the following:

Sign up for a Smart test account on CloudBees Smart Tests.
Create an organization and a workspace for your test suite.
Create an API key for your workspace. This authentication token lets the CLI talk to your CloudBees Smart Tests workspace.

GitHub Actions

The CloudBees Smart Tests record build and test results action integrates CloudBees Smart Tests into your CI with minimal configuration. This action installs the CLI and runs smart-tests record build and smart-tests record tests to send data to CloudBees Smart Tests, which analyzes the test results to improve developer productivity.

For more information, refer to CloudBees Smart Tests record build and test results action.

Example usage

name: Test

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  SMART_TESTS_TOKEN: ${{ secrets.SMART_TESTS_TOKEN }}
  SMART_TESTS_DEBUG: 1
  SMART_TESTS_REPORT_ERROR: 1

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - name: Test
        run: <YOUR TEST COMMAND HERE>
      - name: Record
        uses: launchableinc/record-build-and-test-results-action@v1.0.0
        with:
          build_name: $GITHUB_RUN_ID
          test_runner: <YOUR TEST RUNNER HERE>
          report_path: <PATH TO TEST REPORT XML FILES>
        if: always()▼

Table 1. Test runner values
Test runner	`test_runner` value	Additional steps?
Ant	`ant`
Bazel	`bazel`
Behave	`behave`	Yes
CTest	`ctest`	Yes
cucumber	`cucumber`	Yes
Cypress	`cypress`
GoogleTest	`googletest`	Yes
Go Test	`go-test`	Yes
Gradle	`gradle`
Jest	`jest`	Yes
Maven	`maven`
minitest	`minitest`	Yes
NUnit Console Runner	`nunit`	Yes
pytest	`pytest`	Yes
Robot	`robot`
RSpec	`rspec`	Yes

Instructions for test runners/build tools

The following section provides instructions for configuring test runners or build tools to generate test reports that CloudBees Smart Tests can consume.If your test runner or build tool isn’t listed, use the CLI to send data to CloudBees Smart Tests using one of the following profiles:

Android Debug Bridge (adb)

Currently, the CLI doesn’t have a record tests command for ADB. Use the command for Gradle instead.

Ant

No special steps.

Bazel

No special steps.

Behave

In order to generate reports that CloudBees Smart Tests can consume, add the --junit option to your existing behave command:

# run the tests however you normally do
behave --junit▼

CTest

Run your tests with ctest -T test --no-compress-output. These options ensure test results are written to the Testing directory.

cucumber

Run cucumber with the -f junit option, like this:

bundle exec cucumber -f junit -o reports

(If using JSON, use the CloudBees Smart Tests CLI instead.)

Cypress

No special steps.

GoogleTest

Configure GoogleTest to produce JUnit compatible report files. For more information on how to do this, refer to their documentation.

Command example

# run the tests however you normally do
./my-test --gtest_output=xml:./report/my-test.xml▼

Go Test

In order to generate reports that CloudBees Smart Tests can consume, use go-junit-report to generate a JUnit XML file after running tests:

# install JUnit report formatter
go get -u github.com/jstemmer/go-junit-report

# run the tests however you normally do, then produce a JUnit XML file
go test -v ./... | go-junit-report -set-exit-code > report.xml▼

Gradle

No special steps. Gradle writes test reports to /build//TEST-.xml by default, which the CloudBees Smart Tests CLI consumes. /build//TEST-.xml .

Jest

In order to generate reports that CloudBees Smart Tests can consume, use jest-junit to generate a JUnit XML file after running tests.

# install jest-junit reporter
npm install jest-junit --save-dev
# or
yarn add --dev jest-junit▼

Configure jest-junit to include file paths in reports. Do this using the following environment variables:

Recommended config:

export JEST_JUNIT_CLASSNAME="{classname}"
export JEST_JUNIT_TITLE="{title}"
export JEST_JUNIT_SUITE_NAME="{filepath}"▼

Minimum config:

export JEST_JUNIT_SUITE_NAME="{filepath}"

Add the following lines to package.json. The detail is the jest-junit section.

Recommended config:

"jest-junit": {
  "suiteNameTemplate": "{filepath}",
  "classNameTemplate": "{classname}",
  "titleTemplate": "{title}"
}▼

Minimum config:

"jest-junit": {
  "suiteNameTemplate": "{filepath}"
}▼

Run jest using jest-junit:

# run tests with jest-junit
jest --ci --reporters=default --reporters=jest-junit▼

Maven

CloudBees Smart Tests supports test reports generated using Surefire, the default report plugin for Maven.

Example:

'./**/target/surefire-reports'

The invocation above relies on the CLI to expand globs like **.

minitest

Use minitest-ci to output test results to a file. If storing test results on your CI server, it may already be installed.

NUnit Console Runner

CloudBees Smart Tests CLI accepts NUnit3 style test report XML files produced by NUnit.

pytest

Run tests with the --junit-xml option:

pytest --junit-xml=test-results/results.xml

pytest changed its default test report format from xunit1 to xunit2 in version 6. Unfortunately, the new xunit2 format does not include file paths, which CloudBees Smart Tests needs.

If using pytest 6 or newer, you must also specify junit_family=legacy as the report format. For more information, refer to Deprecations and Removals — pytest documentation for instructions.

Robot

Example output:

output.xml

RSpec

Use rspec_junit_formatter to output test results to a file in RSpec. If storing test results on your CI server, it may already be installed.

Example output:

bundle exec rspec --format RspecJunitFormatter --out report/rspec.xml