Supported test frameworks and integrations

The CloudBees Smart Tests CLI includes built-in integrations for test runners and build tools that execute tests built using various frameworks. CloudBees Smart Tests works with any test framework, as long as a supported test runner or build tool is used to run tests. The integrations interface with popular test runners and build tools to send data to CloudBees Smart Tests without needing to write custom scripts.

The following list describes how to use the CLI with each of these frameworks and integrations. If your test runner or build tool is not listed, consider the Other integrations options.

Supported test frameworks

The following list identifies the supported test frameworks and their corresponding test runners or build tools that integrate with CloudBees Smart Tests.

Table 1. Supported test framework and integrations
Test framework	Integrations
Appium	Android Debug Bridge (adb), Gradle, Maven, pytest
Cucumber	cucumber
GoogleTest	GoogleTest
Jest	Jest
JUnit	Gradle, Maven
minitest	minitest
nose	pytest
NUnit	dotnet test
prove	prove for Perl
pytest	pytest
Robot	Robot
RSpec	RSpec
Selenium	Ant, Gradle, Maven, pytest
TestNG	Gradle, Maven

Android Compatibility Test Suite (CTS)

CloudBees Smart Tests interfaces with Android-CTS via the CloudBees Smart Tests CLI.

Android-CTS recording test results

After running tests, point the CLI to your test report file(s):

smart-tests record tests cts --no-build android-cts/results/*/.xml

Android-CTS subsetting

This profile only supports Zero Input Subsetting.

Android Debug Bridge (adb)

CloudBees Smart Tests interfaces with ADB via the CloudBees Smart Tests CLI.

ADB recording test results

Currently, the CLI doesn’t have a record tests command for ADB. Use the command for Gradle instead.

ADB subsetting

First, request a subset of tests from your entire test suite. Then, pass this list to adb to run.

Request a subset of tests

Find the adb command used to run tests in your CI script. These commands will go before that command.

Duplicate the adb command you normally use to run tests and add the -e log true option.
Then, output the result to a text file. For example:

adb shell am instrument <OPTIONS> -e log true com.yourdomain.test/androidx.test.runner.AndroidJUnitRunner > test_list.txt

This command outputs the full list of tests that would normally run (without actually running them) to a file called test_list.txt.
Pipe the file you just created into smart-tests subset to request a subset from the full list.

cat test_list.txt | smart-tests subset adb --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
  
  This creates a file called smart-tests-subset.txt. This file contains a list of test classes formatted for passing into your normal adb command, shown next.

Run a subset of tests

Now the subset of tests can only be run by adding the -e class $(cat smart-tests-subset.txt) option to your standard adb command, like this:

adb shell am instrument <OPTIONS> -e class $(cat smart-tests-subset.txt) com.yourdomain.test/androidx.test.runner.AndroidJUnitRunner

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
adb shell am instrument <OPTIONS> com.yourdomain.test/androidx.test.runner.AndroidJUnitRunner▼

And the flow after:

# generate the complete list of tests in your suite
adb shell am instrument <OPTIONS> -e log true com.yourdomain.test/androidx.test.runner.AndroidJUnitRunner > test_list.txt
# request a subset from the full list
cat test_list.txt | smart-tests subset adb --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
# run the results of the subset request
adb shell am instrument <OPTIONS> -e class $(cat smart-tests-subset.txt) com.yourdomain.test/androidx.test.runner.AndroidJUnitRunner▼

Ant

CloudBees Smart Tests interfaces with Ant via the CloudBees Smart Tests CLI.

Ant recording test results

After running tests, point the CLI to your test report file(s):

smart-tests record tests ant --no-build <PATH(S) TO JUNIT XML FILE(S)>
Then, follow the instructions to ensure record tests always runs.

Ant subsetting

First, request a subset of tests from your complete suite. Then, pass this list into your build.xml file to limit what Ant runs.

Request a subset of tests

Find the ant command used to run tests in your CI script.
Before that command, add the smart-tests subset command to request a subset of tests from your full test suite:

smart-tests subset ant --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> <PATH TO SOURCE> > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
- Set <PATH TO SOURCE> to the path(s) containing your test files. The CLI will look in those path(s) and generate the full list of tests that would normally run. The subset service divides this whole list into a subset and a remainder.
  
  This creates a file called smart-tests-subset.txt. This file contains a list of test classes formatted for passing into your build.xml file, shown next.

Run a subset of tests

Separately, update your build.xml file to use smart-tests-subset.txt:

<project>
  …
  <target name="check-launchable">
    <available file="smart-tests-subset.txt" property="smart-tests"/>
  </target>

  <target name="junit" depends="jar,check-launchable">
    <mkdir dir="${report.dir}"/>
    <junit printsummary="yes">
      <classpath>
          <path refid="classpath"/>
          <path refid="application"/>
      </classpath>

      <formatter type="xml"/>

      <batchtest fork="yes" todir="${report.dir}">
        <fileset dir="${src.dir}" >
          <includesfile name="smart-tests-subset.txt" if="${smart-tests}" />
          <include name="**/*Test.java" unless="${smart-tests}" />
        </fileset>
      </batchtest>
    </junit>
  </target>
  …
</project>▼

Run your tests as normal, such as:

ant junit <OPTIONS>

Bazel

CloudBees Smart Tests interfaces with Bazel via the CloudBees Smart Tests CLI.

Bazel recording test results

After running tests, point the CLI to your test report files to record test results:

smart-tests record tests bazel --no-build
Then, follow the instructions to Ensure record tests always runs.

Bazel subsetting

First, request a subset of tests from your entire test suite. Then, pass this list to Bazel to run.

Request a subset of tests

Find the bazel command used to run tests in your CI script. These commands will go before that command.

Run bazel query and output the result to a text file. For example:

bazel query 'tests(//…)' > test_list.txt

This command outputs the complete list of test targets that typically run (without running them) to a file called test_list.txt. The subset service will divide this list into a subset and a remainder.
Pipe the file you just created into smart-tests subset to request a subset from the full list.

cat test_list.txt | smart-tests subset bazel --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
  
  This creates a file called smart-tests-subset.txt that you can pass into Bazel.

Run a subset of tests

Append the list of tests to run to your existing command, such as:

bazel test $(cat smart-tests-subset.txt)

Summary

In summary, here’s the flow before:

# your standard command to run tests looks something like this
bazel test▼

And the flow after:

# generate the full list
bazel query 'tests(//...)' > test_list.txt
# request a subset
cat test_list.txt | smart-tests subset bazel --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
# run the results of the subset request
bazel test $(cat smart-tests-subset.txt)▼

Behave

CloudBees Smart Tests interfaces with Behave via the CloudBees Smart Tests CLI.

Behave recording test results

Generate reports that CloudBees Smart Tests can consume, add the --junit option to your existing behave command:
```
# run the tests however you usually do
behave --junit▼
```
After running tests, point the CLI to your test report files to record test results:

smart-tests record tests behave --no-build ./reports/*.xml
Then, follow the instructions to Ensure record tests always runs.

Behave subsetting

First, request a subset of tests from your entire test suite. Then, pass this list to Behave to run.

Request a subset of tests

Find the behave command used to run tests in your CI script. These commands will go before that command.

Run find ./features/ (or a similar command for your environment) and output the result to a text file. For example:

find ./features/ > test_list.txt

This command writes the list of test files that typically run (without actually running them) to test_list.txt. The subset service will divide this list into a subset and a remainder.
Pipe the file you just created into smart-tests subset to request a subset from the full list.

cat test_list.txt | smart-tests subset behave --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
  
  This creates a file called smart-tests-subset.txt that you can pass into Behave.

Run a subset of tests

To run a subset, run behave with the -i option and pass in the subset list. For example:

behave <OPTIONS> -i "$(cat smart-tests-subset.txt)"

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
behave <OPTIONS>▼

And the flow after:

# generate the full list
find ./features/ > test_list.txt
# request a subset
cat test_list.txt | smart-tests subset behave --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
# run the results of the subset request
behave <OPTIONS> -i "$(cat smart-tests-subset.txt)"▼

CTest

CloudBees Smart Tests interfaces with CTest via the CloudBees Smart Tests CLI.

CTest recording test results

Run your tests with ctest -T test --no-compress-output . These options ensure test results are written to the Testing directory.
After running tests, point the CLI to your test report files to record test results:

smart-tests record tests ctest --no-build "Testing/**/Test.xml"
Then, follow the instructions to Ensure record tests always runs.

CTest subsetting

First, request a subset of tests from your entire test suite. Then, pass this list into CTest to run.

Request a subset of tests

Find the ctest command used to run tests in your CI script. These commands will go before that command.

Run ctest with the --show-only option and output the result to a JSON file. For example:

ctest <OPTIONS> --show-only=json-v1 > test_list.json

This command creates the complete list of test files that typically run (without actually running them) to a file called test_list.json. The subset service will divide this list into a subset and a remainder list.
Pass the file you just created into smart-tests subset to request a subset from the full list.

smart-tests subset ctest --session <SESSION_NAME> --confidence <TARGET> --output-regex-files --output-regex-files-dir=subsets test_list.json
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
- The --output-regex-files instructs CLI to write the regular expression for the subset tests into the directory specified in --output-regex-files-dir.
  
  This creates files under the subsets directory. subset_N are the files that contain regular expressions of the chosen subset of tests. If you use the --rest option, rest_N will contain the non-chosen tests.

Run a subset of tests

Run ctest for each subset output file:

for file in subset/subset_*; do
  ctest <OPTIONS> -T test --no-compress-output -R "$(cat "$file")"
done▼

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
ctest <OPTIONS> -T test --no-compress-output▼

And the flow after:

# generate the full list that would normally run
ctest --show-only=json-v1 > test_list.json
# request a subset
smart-tests subset ctest --session <SESSION_NAME> --confidence <TARGET> --output-regex-files --output-regex-files-dir=subsets test_list.json
# run the results of the subset request
for file in subset/subset_*; do
  ctest <OPTIONS> -T test --no-compress-output -R "$(cat "$file")"
done▼

cucumber

CloudBees Smart Tests interfaces with cucumber via the CloudBees Smart Tests CLI.

cucumber recording test results

CloudBees Smart Tests supports JSON and JUnit XML reports generated by cucumber (using -f junit or -f json . Follow the instructions for whichever one being used. JUnit XML is the default.

If you run cucumber with the -f junit option, like this:

bundle exec cucumber -f junit -o reports

After running tests, point the CLI to your test report files to record test results:

smart-tests record tests cucumber --no-build "./reports/*/.xml"
Then, follow the instructions to Ensure record tests always runs.

If you run cucumber with the -f json option, like this:

bundle exec cucumber -f json -o reports

After running tests, point the CLI to your test report files to record test results (note the --json option):

smart-tests record tests cucumber --no-build --json "./reports/*/.json"
Then, follow the instructions to Ensure record tests always runs.

If you receive a warning message such as Cannot find test file Test-feature-example.xml set the project’s root directory path with the --base option:

smart-tests record tests cucumber --no-build --base /example/project "/example/project/reports/*/.xml"

cucumber subsetting

First, request a subset of tests from your entire test suite. Then, pass this list to cucumber to run.

Request a subset of tests

Find the bundle exec cucumber command used to run tests in your CI script.
Before that command, add the smart-tests subset command to request a subset of tests from your full test suite:

smart-tests subset cucumber --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> --base $(pwd) <PATH TO .feature FILES> > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
- Include the --base $(pwd) option (or equivalent) before cucumber.
- Set <PATH TO .feature FILES> to the glob expression representing your .feature files, for example, features/*/.feature. The CLI will look in those path(s) and generate the complete list of tests that would typically run. The subset service divides this list into a subset and a remainder list.
  
  This creates a file called smart-tests-subset.txt. This file contains a list of test files formatted for passing into cucumber, shown next.

Run a subset of tests

Append the subset list to your bundle exec cucumber command to run a subset. For example:

bundle exec cucumber -f junit -o reports <OPTIONS> $(cat smart-tests-subset.txt)

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
bundle exec cucumber -f junit -o reports <OPTIONS>▼

And the flow after:

# request a subset from all features that would typically run
smart-tests subset cucumber --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> <PATH TO .feature FILES> > smart-tests-subset.txt
# run the results of the subset request
bundle exec cucumber -f junit -o reports <OPTIONS> $(cat smart-tests-subset.txt)▼

Cypress

CloudBees Smart Tests interfaces with Cypress via the CloudBees Smart Tests CLI.

Cypress recording test results

After running tests, point the CLI to your test report files to record test results:

smart-tests record tests cypress --no-build ./report/*.xml
Then, follow the instructions to Ensure record tests always runs.

Cypress subsetting

Find the cypress run command used to run tests in your CI script. These commands will go before that command.

Run find ./cypress/integration -type f (or a similar command for your platform) and output the result to a text file. For example:

find ./cypress/integration -type f > test_list.txt

This command writes the complete list of test files that typically run (without actually running them) to test_list.txt. The subset service will divide this list into a subset and a remainder list.
Pipe the file you just created into smart-tests subset to request a subset from the full list.

cat test_list.txt | smart-tests subset cypress --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
  
  This creates a file called smart-tests-subset.txt that you can pass into Cypress.

Run a subset of tests

To run a subset, use the --spec option with the subset list text file. For example:

cypress run --reporter junit <OPTIONS> --spec "$(cat smart-tests-subset.txt)"

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
cypress run --reporter junit <OPTIONS>▼

And the flow after:

# generate the complete list that would typically run
find ./cypress/integration -type f > test_list.txt
# request a subset from all features that would typically run
cat test_list.txt | smart-tests subset cypress --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
# run the results of the subset request
cypress run --reporter junit <OPTIONS> --spec "$(cat smart-tests-subset.txt)"▼

dotnet test

CloudBees Smart Tests interfaces with dotnet test via the CloudBees Smart Tests CLI.

dotnet test recording test results

Configure dotnet test to produce NUnit report files. Install the link+https://www.nuget.org/packages/NUnitXml.TestLogger+[NunitXml.TestLogger] package from NuGet via your preferred method, then add the -l option to your dotnet test command to enable the logger:

dotnet test … -l nunit
After running tests, point the CLI to your test report files to record test results:

smart-tests record tests dotnet --no-build "*/.xml"
Then, follow the instructions to Ensure record tests always runs.

dotnet test subsetting

This profile only supports Zero input subsetting.

flutter

CloudBees Smart Tests interfaces with flutter via the CloudBees Smart Tests CLI.

flutter recording test results

Run tests with the --machine option and produce a test report as JSON (NDJSON) format.
```
flutter test --machine > report.json▼
```

After running tests, point the CLI to your test report file to record test results:

smart-tests record tests \
  --session $(cat smart-tests-session.txt) \
  flutter \
  report.json▼

flutter subsetting

First, request a subset of tests from your full test suite. Then, pass this list into Flutter to run.

Request a subset of tests

First, find the flutter test command used to run tests in your CI script.

smart-tests subset --session <SESSION NAME> <OPTIMIZATION TARGET OPTION> flutter <PATH TO SOURCE>▼

Run a subset of tests

flutter test $(cat smart-tests-subset.txt) --machine > report.json▼

Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.

Summary

In summary, here’s the flow before:

flutter test --machine > report.json▼

And the flow after:

# request a subset
smart-tests subset --session <SESSION NAME> <OPTIMIZATION TARGET OPTION> flutter test/**/*.dart

# run the results of the subset request
flutter test $(cat smart-tests-subset.txt) --machine > report.json▼

GoogleTest

CloudBees Smart Tests interfaces with GoogleTest via the CloudBees Smart Tests CLI.

GoogleTest recording test results

Configure GoogleTest to produce JUnit-compatible report files. For more information, refer to the GoogleTest documentation. You end up with a command something like this:
```
# run the tests however you normally do
./my-test --gtest_output=xml:./report/my-test.xml▼
```
After running tests, point the CLI to your test report files to record test results:

smart-tests record tests googletest --no-build ./report
Then, follow the instructions to Ensure record tests always runs.

GoogleTest subsetting

Find the GoogleTest command used to run tests in your CI script. These commands will go before that command.

Invoke GoogleTest with the --gtest_list_tests option and output the result to a text file. For example:

./my-test <OPTIONS> --gtest_list_tests > test_list.txt

This command outputs the complete list of tests that normally run (without running them) to a file called test_list.txt. The subset service will divide this list into a subset and a remainder list.
Pipe the file you just created into smart-tests subset to request a subset from the full list.

cat test_list.txt | smart-tests subset googletest --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
- Refer to Options for setting <SESSION> and <OPTIMIZATION TARGET OPTION>.
  
  This creates a file called smart-tests-subset.txt that you can pass into GoogleTest.

Run a subset of tests

Add the --gtest_filter option to your existing command, such as:

./my-test <OPTIONS> --gtest_filter="$(cat smart-tests-subset.txt)"

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
./my-test <OPTIONS>▼

And the flow after:

# generate the full list
./my-test <OPTIONS> --gtest_list_tests > test_list.txt
# request a subset
cat test_list.txt | smart-tests subset googletest --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
# run the results of the subset request
./my-test <OPTIONS> --gtest_filter="$(cat smart-tests-subset.txt)"▼

Go Test

CloudBees Smart Tests interfaces with Go Test via the CloudBees Smart Tests CLI.

Go Test recording test results

Generate reports that CloudBees Smart Tests can consume, use go-junit-report to generate a JUnit XML file after you run tests:

# install JUnit report formatter
go get -u github.com/jstemmer/go-junit-report

# run the tests however you usually do, then produce a JUnit XML file
go test -v ./... | go-junit-report -set-exit-code > report.xml▼

Then point the CLI to your test report file to record test results:

smart-tests record tests go-test --no-build report.xml
Then, follow the instructions to Ensure record tests always runs.

Go Test subsetting

Find the go test command used to run tests in your CI script. These commands will go before that command.

Duplicate the go test command you normally use to run tests and add the -list option. Then, output the result to a text file. For example:

go test <OPTIONS> -list="Test|Example" ./… > test_list.txt

This command outputs the complete list of tests that normally run (without running them) to a file called test_list.txt. The subset service will divide this full list into a subset and a remainder.
Pipe the file you just created into smart-tests subset to request a subset from the full list.

cat test_list.txt | smart-tests subset go-test --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
  
  This creates a file called smart-tests-subset.txt that you can pass into go test.

Run a subset of tests

Add the -run option to your existing command, such as:

go test <OPTIONS> -run $(cat smart-tests-subset.txt) ./… | go-junit-report > report.xml

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
go test <OPTIONS> ./...▼

And the flow after:

# generate the full list
go test <OPTIONS> -list="Test|Example" ./... > test_list.txt
# request a subset
cat test_list.txt | smart-tests subset go-test --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
# run the results of the subset request
go test <OPTIONS> -run $(cat smart-tests-subset.txt) ./... | go-junit-report > report.xml▼

Gradle

CloudBees Smart Tests interfaces with Gradle via the CloudBees Smart Tests CLI.

Gradle recording test results

After running tests, point the CLI to your test report files to record test results:

smart-tests record tests gradle --no-build ./build/test-results/test/
Then, follow the instructions to Ensure record tests always runs.

By default, Gradle’s report files are saved to build/test-results/test/ , but that might differ depending on how your Gradle project is configured.
You can specify multiple directories if you do a multi-project build.
A dedicated Gradle task to list all report directories might be convenient for a large project. For more information, refer to the Gradle documentation.
Alternatively, you can specify a glob pattern for directories or individual test report files (this pattern might already be set in your pipeline script for easy copy-pasting). For example, gradle /build//TEST-*.xml .

Gradle subsetting

First, request a subset of tests from your full test suite. Then, pass this list to Gradle.

Request a subset of tests

Find the gradle command used to run tests in your CI script.
Before that command, add the smart-tests subset command to request a subset of tests from your full test suite:

smart-tests subset gradle --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> <PATH TO SOURCE> > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
- Set <PATH TO SOURCE> to the path(s) containing your test files, for example, project1/src/test/java project2/src/test/java. The CLI will look in those path(s) and generate the full list of tests that would normally run. The subset service divides this full list into a subset and a remainder.
  
  This creates a file called smart-tests-subset.txt. This file contains a list of test classes formatted for passing into Gradle, like this:
  
  --tests MyTestClass1 --tests MyTestClass2 …

Run a subset of tests

Pass this file into your existing command, as shown below:

gradle test <OPTIONS> $(cat smart-tests-subset.txt)
# equivalent to gradle test <OPTIONS> --tests MyTestClass1 --tests MyTestClass2 ...▼

The Gradle plugin for Android requires a different command, because the built-in test task does not support the --tests option. Use testDebugUnitTest or testReleaseUnitTest instead:

./gradlew testDebugUnitTest <OPTIONS> $(cat smart-tests-subset.txt)
# or
./gradlew testReleaseUnitTest <OPTIONS> $(cat smart-tests-subset.txt)▼

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
gradle test <OPTIONS>▼

And the flow after:

# request a subset from all tests
smart-tests subset gradle --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> <PATH TO SOURCE> > smart-tests-subset.txt
# run the results of the subset request
gradle test <OPTIONS> $(cat smart-tests-subset.txt)▼

Gradle + TestNG

CloudBees Smart Tests interfaces with Gradle + TestNG via the CloudBees Smart Tests CLI.

Gradle + TestNG recording test results

Currently, the CLI doesn’t have a record tests command for Gradle + TestNG. Use the command for Gradle instead.

Gradle + TestNG subsetting

First, request a subset of tests from your full test suite. Then, pass this list to Gradle.

Request a subset of tests

Find the gradle command used to run tests in your CI script.
Before that command, add the smart-tests subset command to request a subset of tests from your full test suite:

smart-tests subset gradle --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> --bare <PATH TO SOURCE> > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
- Set <PATH TO SOURCE> to the path(s) containing your test files, for example, project1/src/test/java project2/src/test/java. The CLI will look in those path(s) and generate the full list of tests that would normally run. The subset service divides this full list into a subset and a remainder.
- Include the --bare option after gradle.
  
  This creates a file called smart-tests-subset.txt. This file contains a list of test classes formatted for passing into Gradle, like this:
  com.example.FooTest com.example.BarTest ...
  ▼

Run a subset of tests

Add a dependency declaration to build.gradle so that the right subset of tests is executed when TestNG runs:
```
dependencies {
    ...
    testRuntime 'com.launchableinc:launchable-testng:1.2.1'
}▼
```

Export the subset file path as an environment variable before you run gradle test, like shown below.

export SMART_TESTS_SUBSET_FILE_PATH=$PWD/smart-tests-subset.txt
gradle test <OPTIONS>▼

The Gradle plugin for Android requires a different command, because the built-in test task does not support the --tests option. Use testDebugUnitTest or testReleaseUnitTest instead:

export SMART_TESTS_SUBSET_FILE_PATH=$PWD/smart-tests-subset.txt
./gradlew testDebugUnitTest <OPTIONS> $(cat smart-tests-subset.txt)
# or
export SMART_TESTS_SUBSET_FILE_PATH=$PWD/smart-tests-subset.txt
./gradlew testReleaseUnitTest <OPTIONS> $(cat smart-tests-subset.txt)▼

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
gradle test <OPTIONS>▼

And the flow after:

# request a subset from all tests
smart-tests subset gradle --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> --bare <PATH TO SOURCE> > smart-tests-subset.txt
# run the results of the subset request using the `launchable-testng` plugin
export SMART_TESTS_SUBSET_FILE_PATH=$PWD/smart-tests-subset.txt
gradle test <OPTIONS>▼

Jest

CloudBees Smart Tests interfaces with Jest via the CloudBees Smart Tests CLI.

Jest recording test results

Generate reports that CloudBees Smart Tests can consume, use jest-junit to generate a JUnit XML file after you run tests.
```
# install jest-junit reporter
npm install jest-junit --save-dev
# or
yarn add --dev jest-junit▼
```
Configure jest-junit to include file paths in reports.

You can do this using environment variables:
Using environment variables
Using package.json
Recommended config:
export JEST_JUNIT_CLASSNAME="{classname}" export JEST_JUNIT_TITLE="{title}" export JEST_JUNIT_SUITE_NAME="{filepath}"
▼
Minimum config:

export JEST_JUNIT_SUITE_NAME="{filepath}"
Add the following lines to your package.json . The detail is the jest-junit section.

Recommended config:
"jest-junit": { "suiteNameTemplate": "{filepath}", "classNameTemplate": "{classname}", "titleTemplate": "{title}" }
▼
Minimum config:
"jest-junit": { "suiteNameTemplate": "{filepath}" }
▼

Run jest using jest-junit:

# run tests with jest-junit
jest --ci --reporters=default --reporters=jest-junit▼

After running tests, point the CLI to your test report files to record test results:

# send test results to {PRODUCT}
smart-tests record tests jest --no-build your-junit.xml▼

Then, follow the instructions to Ensure record tests always runs.

Jest subsetting

Find the jest command used to run tests in your CI script. These commands will go before that command.

Duplicate the jest command you normally use to run tests and add the --listTests option. Then, output the result to a text file. For example:

jest <OPTIONS> --listTests > test_list.txt

This command creates the full list of test files that would normally run (without actually running them) to a file called test_list.txt. The subset service will divide this full list into a subset and a remainder.
Pipe the file you just created into smart-tests subset to request a subset from the full list.

cat test_list.txt | smart-tests subset jest --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> --base $(pwd) > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
- Include the --base $(pwd) option before jest.
  
  This creates a file called smart-tests-subset.txt that you can pass into Jest.

Run a subset of tests

To run the subset, include the subset list after jest. For example:

jest <OPTIONS> $(cat smart-tests-subset.txt)

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
jest <OPTIONS>▼

And the flow after:

# generate the full list that would normally run
jest <OPTIONS> --listTests > test_list.txt
# request a subset from all features that would normally run
cat test_list.txt | smart-tests subset jest --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> --base $(pwd) > smart-tests-subset.txt
# run the results of the subset request
jest <OPTIONS> $(cat smart-tests-subset.txt)▼

Karma

CloudBees Smart Tests interfaces with Karma via the CloudBees Smart Tests CLI.

Karma recording test results

Generate reports that CloudBees Smart Tests can consume, use karma-json-reporter to generate JSON test reports. Refer to the linked page for more details, but your karma.conf.js change should look something like this:

module.exports = function (config) {
  config.set({
    ...
    plugins: [
      ...
      require('karma-json-reporter')
    ],
    jsonReporter: {
      outputFile: require('path').join(__dirname, 'test-results.json'),
      stdout: false
    },
    reporters: [..., 'json']
  });
};▼

After running tests, point the CLI to your test report file to record test results:
```
smart-tests record tests karma --session $(cat session.txt) test-results.json▼
```
Then, follow the instructions to Ensure record tests always runs.

Karma subsetting

Before invoking your tests with Karma, run the subset karma command, which expects a list of test files as input, for example:

find src -name "*.spec.ts" -o -name "*.spec.js" | \
smart-tests subset --session <SESSIONID> <OPTIMIZATION TARGET OPTION> karma > subset.txt▼

Refer to Options for setting <SESSIONID> and <OPTIMIZATION TARGET OPTION>.

subset.txt will contain a list of test files that are selected as a subset, which you then pass to the test runner invocation. If you run Karma via ng test, you can use the --with ng option so that the output will be in the form of --include path/to/some.spec.ts, which is expected by ng test.

find ... | smart-tests ... karma --with ng > subset.txt
ng test $(cat subset.txt)▼

Maven

CloudBees Smart Tests interfaces with Maven via the CloudBees Smart Tests CLI.

Maven recording test results

CloudBees Smart Tests supports test reports generated using Surefire , the default report plugin for Maven.

After running tests, point the CLI to your test report files to record test results:

smart-tests record tests maven --no-build "./**/target/surefire-reports"

You can pass multiple directories/patterns, e.g. _ smart-tests record tests maven --no-build '<pattern/directory 1>' '<pattern/directory 2>' _.
Then, follow the instructions to Ensure record tests always runs.

Maven subsetting

Find the mvn test command used to run tests in your CI script. These commands will go before that command.

Duplicate the mvn test command you normally use to run tests, but change test to test-compile. For example:

mvn test-compile <OPTIONS>

This command creates .lst files that list the test classes that would normally run (without running them). The subset service combines these and divides the full list into a subset and a remainder.
Run smart-tests subset to request a subset from the full list.

smart-tests subset maven --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> --test-compile-created-file <(find . -path '*/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst' -exec cat {} \;) > smart-tests-subset.txt
- Refer to Options for how to set <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
- The <(find… section combines the .lst files across your projects into a single file for processing. You might need to change this for your platform.
  
  This creates a file called smart-tests-subset.txt that you can pass into Maven.

Run a subset of tests

To run the subset, use the -Dsurefire.includesFile option. For example:

mvn test <OPTIONS> -Dsurefire.includesFile=$PWD/smart-tests-subset.txt

Summary

In summary, here’s the flow before:

# your normal command to run tests looks something like this
mvn test <OPTIONS>▼

And the flow after:

# generate the full list(s) that would normally run
mvn test-compile <OPTIONS>
# request a subset from all features that would normally run
smart-tests subset maven --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> --test-compile-created-file <(find . -path '*/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst' -exec cat {} \;) > smart-tests-subset.txt
# run the results of the subset request
mvn test <OPTIONS> -Dsurefire.includesFile=$PWD/smart-tests-subset.txt▼

Maven + TestNG

CloudBees Smart Tests interfaces with Maven via the CloudBees Smart Tests CLI.

Maven + TestNG recording test results

Currently, the CLI doesn’t have a record tests command for Maven + TestNG. Use the command for Maven instead.

Maven + TestNG subsetting

Find the mvn test command used to run tests in your CI script. These commands will go before that command.

Duplicate the mvn test command you normally use to run tests, but change test to test-compile. For example:

mvn test-compile <OPTIONS>

This command creates .lst files that list the test classes that would normally run (without running them). The subset service combines these and divides the full list into a subset and a remainder.
Run smart-tests subset to request a subset from the full list.

smart-tests subset maven --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> --test-compile-created-file <(find . -path '*/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst' -exec cat {} \;) > smart-tests-subset.txt
- Refer to Options for how to set <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
- The <(find… section combines the .lst files across your projects into a single file for processing.
  
  This creates a file called smart-tests-subset.txt that you can pass into Maven.

Run a subset of tests

Modify your pom.xml so that it includes CloudBees Smart Tests TestNG integration as a test scope dependency:

<dependency>
  <groupId>com.launchableinc</groupId>
  <artifactId>launchable-testng</artifactId>
  <version>1.2.1</version>
  <scope>test</scope>
</dependency>▼

Export the subset file path as an environment variable before you run mvn test, like shown below.
```
export SMART_TESTS_SUBSET_FILE_PATH=$PWD/smart-tests-subset.txt
mvn test <OPTIONS>▼
```

Summary

In summary, here’s the flow before:

# your normal command to run tests looks something like this
mvn test <OPTIONS>▼

And the flow after:

# generate the full list of tests
mvn test-compile <OPTIONS>
# request a subset from all tests
smart-tests subset maven --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> --test-compile-created-file <(find . -path '*/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst' -exec cat {} \;) > smart-tests-subset.txt
# run the results of the subset request using the `launchable-testng` plugin
export SMART_TESTS_SUBSET_FILE_PATH=$PWD/smart-tests-subset.txt
mvn test <OPTIONS>▼

minitest

CloudBees Smart Tests interfaces with minitest via the CloudBees Smart Tests CLI.

minitest recording test results

Use minitest-ci to output test results to a file. It may already be installed if you store your test results on your CI server.
After running tests, point the CLI to your test report files to record test results:

smart-tests record tests minitest --no-build "$CIRCLE_TEST_REPORTS/reports"
Then, follow the instructions to Ensure record tests always runs.

minitest subsetting

First, request a subset of tests from your full test suite. Then, pass this list into minitest to run.

Request a subset of tests

Find the bundle exec rails test command used to run tests in your CI script.
Before that command, add the smart-tests subset command to request a subset of tests from your full test suite:

smart-tests subset minitest --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> <PATH TO .rb FILES> > smart-tests-subset.txt
- Refer to Options for how to set <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
- Set <PATH TO .rb FILES> to the glob expression representing your .rb test files (for example, test/*/.rb). The CLI will look in those path(s) and generate the full list of tests that would normally run. The subset service divides this full list into a subset and a remainder.
  
  This creates a file called smart-tests-subset.txt. This file contains a list of tests formatted for passing into minitest.

Run a subset of tests

To run a subset, pass the subset list into bundle exec rails test . For example:

bundle exec rails test <OPTIONS> $(cat smart-tests-subset.txt)

Summary

In summary, here’s the flow before:

# your normal command to run tests looks something like this
bundle exec rails test <OPTIONS>▼

And the flow after:

# request a subset of your existing test suite
smart-tests subset minitest --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> <PATH TO .rb FILES> > smart-tests-subset.txt
# run the results of the subset request
bundle exec rails test <OPTIONS> $(cat smart-tests-subset.txt)▼

NUnit Console Runner

CloudBees Smart Tests interfaces with NUnit Console Runner via the CloudBees Smart Tests CLI.

NUnit Console Runner recording test results

CloudBees Smart Tests CLI accepts NUnit3 style test report XML files produced by NUnit.

After running tests, point the CLI to your test report file(s) to record test results:

smart-tests record tests nunit --no-build path/to/TestResult.xml
Then, follow the instructions to Ensure record tests always runs.

NUnit Console Runner subsetting

First, request a subset of tests from your full test suite. Then, pass this list into nunit3-console to run.

Request a subset of tests

Find the nunit3-console command used to run tests in your CI script. These commands will go before that command.

Duplicate the nunit3-console command you normally use to run tests, and add the --explore option. For example:

nunit3-console <OPTIONS> --explore=test_list.xml path/to/myassembly.dll

This command writes the full list of tests that normally run (without running them) to test_list.xml.
Pass the file you just created into smart-tests subset to request a subset from the full list.

smart-tests subset nunit --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> test_list.xml > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
  
  This creates a file called smart-tests-subset.txt. This file contains a list of test classes formatted for passing into your normal adb command, shown next.

If you want to subset tests across multiple DLLs (for example, if multiple DLLs are combined into a logical 'suite'), run nunit3-console --explore… once for each DLL, then pass all the files into smart-tests subset, such as:

nunit3-console <OPTIONS> --explore=myassembly1.xml path/to/myassembly1.dll
nunit3-console <OPTIONS> --explore=myassembly2.xml path/to/myassembly2.dll
nunit3-console <OPTIONS> --explore=myassembly3.xml path/to/myassembly3.dll

smart-tests subset nunit --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> myassembly1.xml myassembly2.xml myassembly3.xml > smart-tests-subset.txt▼

Run a subset of tests

Now you can run the subset of tests by adding the --testlist option to your normal nunit3-console command, like this:

nunit3-console <OPTIONS> --testlist=smart-tests-subset.txt path/to/myassembly.dll [path/to/myassembly2.dll] [path/to/myassembly3.dll]

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
nunit3-console <OPTIONS> path/to/myassembly.dll▼

And the flow after:

# generate the full list of tests in your suite
nunit3-console <OPTIONS> --explore=test_list.xml path/to/myassembly.dll
# request a subset from the full list
smart-tests subset nunit --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> test_list.xml > smart-tests-subset.txt
# run the results of the subset request
nunit3-console <OPTIONS> --testlist=smart-tests-subset.txt path/to/myassembly.dll [path/to/myassembly2.dll] [path/to/myassembly3.dll]▼

Playwright

CloudBees Smart Tests interfaces with Playwright via the CloudBees Smart Tests CLI.

Playwright recording test results

CloudBees Smart Tests supports JSON and JUnit XML reports generated by Playwright official reporters (using --json).

Follow the instructions for whichever one being used. JUnit XML is the default but CloudBees Smart Tests recommends to use JSON reports and option. Because JUnit XML reports will consolidate multiple same test case results into one.

First, run tests with the --reporter=json option:

PLAYWRIGHT_JSON_OUTPUT_NAME=results.json npx playwright test --reporter=json

or pass options to the configuration file directly:

import { defineConfig } from '@playwright/test';

export default defineConfig({
reporter: [['json', { outputFile: 'results.json' }]],
});▼

For more information, refer to Playwright JSON reporter documentation.

After running tests, point the CLI to your test report file(s) to record tests results

smart-tests record tests playwright --no-build --json results.json
Then, follow the instructions to Ensure record tests always runs.

If you use JUnit Reporter to generate reports, like this:

PLAYWRIGHT_JUNIT_OUTPUT_NAME=results.xml npx playwright test --reporter=junit

or pass options to the configuration file directly

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [['junit', { outputFile: 'results.xml' }]],
});▼

After running tests, point the CLI to your test report files(s) to record test results:

smart-tests record tests playwright --no-build results.xml
Then, follow the instructions to Ensure record tests always runs.

Playwright subsetting

First, request a subset of tests from your entire test suite. Then, pass this list to Playwright to run.

Request a subset of tests

Find the playwright test command used to run tests in your CI script.

List the result in a text file. For example:

find tests/*.spec.ts > test_list.txt

This command outputs the complete list of test targets that typically run (without running them) to a file called test_list.txt. The subset service will divide this list into a subset and a remainder.
Pipe the file you just created into smart-tests subset to request a subset from the full list.

cat test_list.txt | smart-tests subset playwright --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.

Run a subset of tests

Append the list of tests to run on your existing command, such as:

playwright test $(cat smart-tests-subset.txt)

Summary

In summary, here’s the flow before:

playwright test ./tests

And the flow after:

# generate the test list
find ./tests/*.spec.ts > test_list.txt
# request a subset
cat test_list.txt | smart-tests subset playwright --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
# run the results of the subset request
playwright test $(cat smart-tests-subset.txt)▼

prove for Perl

CloudBees Smart Tests interfaces with prove for Perl via the CloudBees Smart Tests CLI.

prove for Perl recording test results

CloudBees Smart Tests supports JUnit XML report generated by TAP::Formatter::JUnit.

Follow the instructions for generating a report.

prove -Ilib --formatter TAP::Formatter::JUnit-r t

The command will generate a JUnit XML report with the name junit_output.xml .
After running tests, point the CLI to your test report file to record test results:

smart-tests record tests prove --no-build junit_output.xml
Then, follow the instructions to Ensure record tests always runs.

prove for Perl subsetting

Find the prove command used to run tests in your CI script. These commands will go before that command.

Pipe the test files you have into smart-tests subset to request a subset from the full list.
```
# Assuming your test directory is `./t`.
find ./t -name '*.t' | smart-tests subset prove --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt▼
```
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
  
  This creates a file called smart-tests-subset.txt. This file contains a list of tests formatted for passing into your normal prove command, shown next.

Run a subset of tests

Now you can run the subset of tests by passing the smart-tests-subset.txt file into prove, like this:

# You must pass the environment variable JUNIT_NAME_MANGLE=none to generate the JUnit XML report in {PRODUCT}'s supported format.
export JUNIT_NAME_MANGLE=none
prove <OPTIONS> -Ilib --harness TAP::Harness::JUnit -r $(cat smart-tests-subset.txt)▼

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
prove <OPTIONS> -Ilib --harness TAP::Harness::JUnit -r t▼

And the flow after:

# request a subset from the full list
find ./t -name '*.t' | smart-tests subset prove --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
# run the results of the subset request
export JUNIT_NAME_MANGLE=none
prove <OPTIONS> -Ilib --harness TAP::Harness::JUnit -r $(cat smart-tests-subset.txt)▼

pytest

CloudBees Smart Tests interfaces with pytest via the CloudBees Smart Tests CLI.

pytest recording test results

CloudBees Smart Tests supports JSON and JUnit XML reports generated by pytest (using --junit-xml or --report-log ).

Follow the instructions for whichever one being used. JUnit XML is the default.

First, run tests with the --junit-xml option:

pytest --junit-xml=test-results/results.xml

pytest changed its default test report format from xunit1 to xunit2 in version 6. Unfortunately, the new xunit2 format does not include file paths, which CloudBees Smart Tests needs.

Therefore, if using pytest 6 or newer, you must also specify junit_family=legacy as the report format. For more information, refer to Deprecations and Removals — pytest documentation.

After running tests, point the CLI to your test report file(s) to record test results (note the --json option):

smart-tests record tests pytest --no-build ./test-results/
Then, follow the instructions to Ensure record tests always runs.

If you use the pytest-dev/pytest-reportlog plugin to generate reports, like this:

pytest --report-log=test-results/results.json

After running tests, point the CLI to your test report file(s) to record test results:

smart-tests record tests pytest --no-build --json ./test-results/
Then, follow the instructions to Ensure record tests always runs.

pytest subsetting

Find the pytest command used to run tests in your CI script. These commands will go before that command.

Duplicate the pytest command you normally use to run tests and add the --collect-only and -q options. Then output that to a file. For example:

pytest <OPTIONS> --collect-only -q > test_list.txt

This command writes the full list of tests that normally run (without running them) to test_list.txt.
Pipe the file you just created into smart-tests subset to request a subset from the full list.

cat test_list.txt | smart-tests subset pytest --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
  
  This creates a file called smart-tests-subset.txt. This file contains a list of tests formatted for passing into your normal pytest command, shown next.

Run a subset of tests

Now you can run the subset of tests by passing the smart-tests-subset.txt file into pytest, like this:

pytest <OPTIONS> --junit-xml=test-results/subset.xml $(cat smart-tests-subset.txt)

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
pytest <OPTIONS> --junit-xml=test-results/subset.xml▼

And the flow after:

# generate the full list of tests in your suite
pytest <OPTIONS> --collect-only -q > test_list.txt
# request a subset from the full list
cat test_list.txt | smart-tests subset pytest --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> > smart-tests-subset.txt
# run the results of the subset request
pytest <OPTIONS> --junit-xml=test-results/subset.xml $(cat smart-tests-subset.txt)▼

Robot

CloudBees Smart Tests interfaces with Robot via the CloudBees Smart Tests CLI.

Robot recording test results

After running tests with robot , point the CLI to your test report files to record test results:

smart-tests record tests robot --no-build output.xml
Then, follow the instructions to Ensure record tests always runs.

Robot subsetting

Find the robot command used to run tests in your CI script. These commands will go before that command.

Duplicate the robot command you normally use to run tests, and add the --dryrun and -o options. For example:

robot <OPTIONS> --dryrun -o test_list.xml

This command writes the full list of tests that normally run (without running them) to test_list.xml.
Pass the file you just created into smart-tests subset to request a subset from the full list.

smart-tests subset robot --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> test_list.xml > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
  
  This creates a file called smart-tests-subset.txt. This file contains a list of tests formatted for passing into your normal robot command, shown next.

Run a subset of tests

Now you can run just the subset of tests by passing the smart-tests-subset.txt file into robot, like this:

robot <OPTIONS> $(cat smart-tests-subset.txt) .

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
robot <OPTIONS>▼

And the flow after:

# generate the full list of tests in your suite
robot <OPTIONS> --dryrun -o test_list.xml
# request a subset from the full list
smart-tests subset robot --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> test_list.xml > smart-tests-subset.txt
# run the results of the subset request
robot <OPTIONS> $(cat smart-tests-subset.txt) .▼

RSpec

CloudBees Smart Tests interfaces with RSpec via the CloudBees Smart Tests CLI.

RSpec recording test results

Use rspec_junit_formatter to output test results to a file in RSpec. If you already have a CI server storing your test results, it may already be installed:

bundle exec rspec --format RspecJunitFormatter --out report/rspec.xml
After running tests, point the CLI to your test report files to record test results:

smart-tests record tests rspec --no-build ./report
Then, follow the instructions to Ensure record tests always runs.

RSpec subsetting

First, request a subset of tests from your full test suite. Then, pass this list to RSpec to run.

Request a subset of tests

Find the bundle exec rspec command used to run tests in your CI script.
Before that command, add the smart-tests subset command to request a subset of tests from your full test suite:

smart-tests subset rspec --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> <PATH(S) TO .rb FILES> > smart-tests-subset.txt
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
- Set <PATH TO .rb FILES> to the glob expression representing your .rb test files. For example, spec/**/*_spec.rb. The CLI will look in those path(s) and generate the full list of tests that would normally run. The subset service divides this full list into a subset and a remainder.
  
  This creates a file called smart-tests-subset.txt. This file contains a list of tests formatted for passing into RSpec.

Run a subset of tests

To run a subset, pass the subset list into bundle exec rspec. For example:

bundle exec rspec $(cat smart-tests-subset.txt) --format d --format RspecJunitFormatter --out rspec.xml <OPTIONS>

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
bundle exec rspec --format RspecJunitFormatter --out report/rspec.xml <OPTIONS>▼

And the flow after:

# request a subset of your existing test suite
smart-tests subset rspec --session <SESSION_NAME> <OPTIMIZATION TARGET OPTION> <PATH TO .rb FILES> > smart-tests-subset.txt
# run the results of the subset request
bundle exec rspec $(cat smart-tests-subset.txt) --format d --format RspecJunitFormatter --out rspec.xml <OPTIONS>▼

Ruby test-unit

CloudBees Smart Tests interfaces with Ruby test-unit via the CloudBees Smart Tests CLI.

Ruby test-unit recording test results

Use test-unit-launchable to output test results to a file in test-unit.

ruby test/example_test.rb --runner=launchable --launchable-test-report-json=report.json▼

After running tests, point the CLI to your test report files to record test results:

smart-tests record tests raw --no-build report.json
Then, follow the instructions to Ensure record tests always runs.

Ruby test-unit subsetting

Currently, the CLI does not have a subset command for Ruby test-unit. Use the raw profile instead.

TestNG

If using the TestNG framework, check whether the tests are run with Gradle or Maven. Then, use the appropriate command for the corresponding tool.

vitest

CloudBees Smart Tests interfaces with vitest via the CloudBees Smart Tests CLI.

vitest recording test results

Run tests with the --reporter junit option and the --outputFile option to produce a test report in JUnit format.
```
vitest run --reporter junit --outputFile report.xml▼
```

After running tests, point the CLI to your test report file to record test results:

smart-tests record tests \
  --session $(cat smart-tests-session.txt) \
  vitest \
  report.json▼

vitest subsetting

First, request a subset of tests from your entire test suite. Then, pass this list to Vitest to run.

Request a subset of tests

Find the vitest run test command used to run tests in your CI script.

Duplicate the vitest command you normally use to run list command and add the --filesOnly option. Then, output the result to a text file. For example:
```
vitest list --filesOnly > test_list.txt▼
```
This command outputs the complete list of test targets that would normally run (without actually running them) to a file called test_list.txt. The subset service will divide this full list into a subset and a remainder.
Pipe the file you just created into smart-tests subset to request a subset from the full list:
```
cat test_list.txt | smart-tests subset vitest --session <SESSION NAME> <OPTIMIZATION TARGET OPTION>▼
```
- Refer to Options for setting <SESSION NAME> and <OPTIMIZATION TARGET OPTION>.
  
  This creates a file called smart-tests-subset.txt that you can pass into Vitest.

Run a subset of tests

To run the subset, include the subset list after vitest. For example:

vitest run <OPTIONS> $(cat smart-tests-subset.txt)▼

Summary

In summary, here’s the flow before:

# Your normal command to run tests looks something like this
vitest run test <OPTIONS>▼

And the flow after:

# generate the full list that would normally run
vitest list --filesOnly > test_list.txt

# request a subset from all features that would normally run
cat test_list.txt | smart-tests subset vitest --session <SESSION NAME> <OPTIMIZATION TARGET OPTION>

# run the results of the subset request
vitest run <OPTIONS> $(cat smart-tests-subset.txt)▼

xctest

CloudBees Smart Tests interfaces with xctest via the CloudBees Smart Tests CLI.

xctest recording test results

Generate reports that CloudBees Smart Tests can consume, use xcpretty to generate a JUnit XML file after you run tests:

# install xcpretty
gem install xcpretty

# run the tests however you usually do, then produce a JUnit XML file
xcodebuild test -scheme <SCHEMA> -destination '<DESTINATION>' -parallel-testing-enabled | xcpretty -r junit▼

Point the CLI to your test report file to record test results:

smart-tests record tests \
  --session $(cat smart-tests-session.txt) \
  xctest \
  build/reports/junit.xml▼

Then, follow the instructions to Ensure record tests always runs.

xctest subsetting

This profile only supports Zero Input Subsetting.

Other integrations

If not using one of the test runners above, CloudBees Smart Tests also uses the following options:

file profile for unsupported test runners
raw profile for custom test runners
Request a plugin from Support or your CloudBees representative.