Getting started
Install
The CloudBees Smart Tests CLI is a Python3 package that you can install from PyPI.
Recommended: Using uv (fastest)
We recommend using uv, a fast Python package installer, for the best installation experience.
First, install uv:
curl -LsSf https://astral.sh/uv/install.sh | sh export PATH="$HOME/.local/bin:$PATH"
For other installation methods (including Windows, Github Actions, etc.), see the uv installation guide.
Then install CloudBees Smart Tests CLI:
uv tool install smart-tests
Alternative: Using pip
You can also install the CLI using pip:
pip3 install --user --upgrade smart-tests
This creates a ~/.local/bin/smart-tests executable that should be in your PATH . (See PEP-370 for further details.)
Verify
Then run smart-tests verify in your CI environment to see if you’ve successfully configured the CLI. If it succeeds, you’ll see a message like the one below. If you see an error message, see Troubleshooting .
$ smart-tests verify Organization: <organization> Workspace: <workspace> Proxy: None Platform: 'macOS-12.0.1-x86_64-i386-64bit' Python version: '3.9.9' Java command: 'java' smart-tests version: '1.34.0' Your CLI configuration is successfully verified 🎉
Required Workflow
|
With the smart-tests command update, you must now follow this mandatory sequence:
The session name specified in |
Command: inspect subset
Display the details of a subset request. See /docs/features/predictive-test-selection/#inspecting-subset-details[Subsetting your test runs] for more info.
Usage: smart-tests inspect subset [OPTIONS]
| Option | Description | Required |
|---|---|---|
|
Display JSON format |
No |
|
Subset id |
Yes |
You can use smart-tests inspect subset to inspect the details of a specific subset, including rank and expected duration. This is useful for verifying that you passed the correct tests or test directory path(s) into smart-tests subset .
The output from smart-tests subset includes a tip to run smart-tests inspect subset :
$ smart-tests subset minitest --build 123 --session session-123 --confidence 90% test/*.rb > subset.txt < summary table > Run `smart-tests inspect subset --subset-id 26876` to view full subset details
Running that command will output a table containing a row for each test, including:
-
Rank/order
-
Test identifier
-
Whether the test was included in the subset
-
CloudBees Smart Tests’s estimated duration for the test * Tests with a duration of
.001seconds were not recognized by CloudBees Smart Tests
|
Note that the hierarchy level of the items in the list depends on the test runner in use. For example, since Maven can accept a list of test classes as input, |
Command: record attachment
Attach log files to test session. See Attaching log files for more details.
Usage: smart-tests record attachment [OPTIONS] <ATTACHMENTS>…
| Argument | Description | Required |
|---|---|---|
|
Attachment files to upload |
Yes |
| Option | Description | Required |
|---|---|---|
|
Session ID obtained by calling 'smart-tests record session'. It also accepts '@path/to/file' if the session ID is stored in a file |
Yes |
Command: record commit
Sends commit details to CloudBees Smart Tests. Records multiple commits from repo(s).
Usage: smart-tests record commit [OPTIONS]
| Option | Description | Required |
|---|---|---|
|
Import from the git-log output |
No |
|
The maximum number of days to collect commits retroactively |
No |
|
Repository name |
No |
|
Repository path |
No |
Commit collection happens automatically as a part of record build , so normally this command need not be invoked separately. It’s only used for /docs/sending-data-to-smart-tests/using-the-smart-tests-cli/recording-builds-with-the-smart-tests-cli/recording-builds-from-multiple-repositories/#multiple-repositories-builtdeployed-separately-then-tested-together-eg-microservices[Multiple repositories built/deployed separately and then tested together (e.g., microservices)] .
--import-git-log-output option
Related to Running under restricted networks .
If the --import-git-log-output option is used, it reads the specified file for the commit data instead of reading the commits from the repository specified by --source . The input file should contain the output of this Git command:
git log --pretty='format:{"commit": "%H", "parents": "%P", "authorEmail": "%ae", "authorTime": "%aI", "committerEmail": "%ce", "committerTime": "%cI"}' --numstat
Command: record build
Creates a record of a Build in CloudBees Smart Tests.
Usage: smart-tests record build [OPTIONS]
| Option | Description | Required |
|---|---|---|
|
Set branch name. A branch is a set of test sessions grouped and this option value will be used for a branch name. |
No |
|
Build name |
Yes |
|
Set repository name and commit hash when you use --no-commit-collection option (can be specified multiple times) |
No |
|
Set external link of a title and url (can be specified multiple times) |
No |
|
The maximum number of days to collect commits retroactively |
No |
|
Do not collect commit data. This is useful if the repository is a shallow clone and the RevWalk is not possible. The commit data must be collected with a separate fully-cloned repository. |
No |
|
Stop collecting information from Git Submodules |
No |
|
Set repository name and branch name when you use --no-commit-collection option. Please use the same repository name with a commit option (can be specified multiple times) |
No |
|
Path to local Git workspace, optionally prefixed by a label. like --source path/to/ws or --source main=path/to/ws (can be specified multiple times) |
No |
|
Used to overwrite the build time when importing historical data. Note: Format must be |
No |
The act of recording a build teaches CloudBees Smart Tests that the specified set of commits has turned into a build and that the given name henceforth identifies this build. This forms the basis of how CloudBees Smart Tests calculates the changes.
Conceptually, a build is a collection of Git repositories, each at a specific commit. REPO_NAME identifies each repository contributing to a build, and it needs to be stable across different builds of the same project. Good examples include:
-
Relative directory paths to the repository from the "workspace root," such as
src/moduleXif they are stable. -
GitHubOrg/GitHubReposlug if your repositories are on GitHub since they are also stable.
Command: record session
|
This command is now mandatory and must be executed after |
Creates a record of a test session in CloudBees Smart Tests.
Usage: smart-tests record session [OPTIONS]
| Option | Description | Required |
|---|---|---|
|
Build name |
Yes |
|
Flavors (can be specified multiple times) |
No |
|
If you want to only send test reports, please use this option |
No |
|
Enable observation mode |
No |
|
Set external link of title and url (can be specified multiple times) |
No |
|
Set test suite name. A test suite is a collection of test sessions. Setting a test suite allows you to manage data over test sessions and lineages. |
Yes |
|
Used to overwrite the session time when importing historical data. Note: Format must be |
No |
This command tells CloudBees Smart Tests that you are about to begin testing a build that was been recorded earlier with the record build command. This command is now mandatory and must be executed after every smart-tests record build command.
The command outputs a string you can save for use in other commands (like smart-tests subset and smart-tests record tests) instead of --build . We suggest saving the value either to an environment variable or to a text file:
# environment variable smart-tests record build --build BUILD_NAME [OPTIONS] export SMART_TESTS_SESSION=$(smart-tests record session --build BUILD_NAME --session SESSION_NAME [OPTIONS]) <run tests> smart-tests record tests TESTRUNNER --session SESSION_NAME [OPTIONS]
# text file smart-tests record build --build BUILD_NAME [OPTIONS] smart-tests record session --build BUILD_NAME --session SESSION_NAME [OPTIONS] <run tests> smart-tests record tests TESTRUNNER --session SESSION_NAME [OPTIONS]
(Otherwise, the command will write a session ID to ~/.config/smart-tests/sessions/{hash}.txt . This location may change in the future, so don’t rely on it.)
Command: record tests
Send test results for the test session to CloudBees Smart Tests.
|
The |
Usage: smart-tests record tests [OPTIONS] <TEST_RUNNER> …
| Argument | Description | Required |
|---|---|---|
|
Yes |
| Option | Description | Required |
|---|---|---|
|
(Advanced) base directory to make test names portable |
No |
|
Grouping name for test results |
No |
|
Do not guess the base path to relativize the test file paths. By default, if the test file paths are absolute file paths, it automatically guesses the repository root directory and relativize the paths. With this option, the command doesn’t do this guess work. If --base-path is specified, the absolute file paths are relativized to the specified path irrelevant to this option. Use it if the guessed base path is incorrect. |
No |
|
Post chunk |
No |
|
Session ID obtained by calling 'smart-tests record session'. It also accepts '@path/to/file' if the session ID is stored in a file |
Yes |
This command reads JUnit (or similar) XML report files produced by test runners and sends them to CloudBees Smart Tests.
Exactly how this command generates the subset and what’s required to do this depends on test runners. For available supported TESTRUNNER , see Integrations
Command: split-subset
Splits an existing subset from CloudBees Smart Tests into chunks. This relates to Replacing static parallel suites with a dynamic parallel subset and Using groups to split subsets .
smart-tests split-subset TESTRUNNER [OPTIONS] …
Intended for use with smart-tests subset with the --split option.
Options for Replacing static parallel suites with a dynamic parallel subset
| Option | Description | Required |
|---|---|---|
|
The portion of the subset to retrieve, e.g |
Yes |
|
Output the remainder of the subset to a file. This is useful for running the "rest of the tests" after you’ve run a subset. |
No |
|
[Beta; Gradle only] Place tests listed in the FILE to belong to the same bin to avoid the tests running simultaneously. |
No |
|
The ID of the subset output from |
Yes |
|
Output a list of tests to exclude instead of a list to include. See Zero Input Subsetting. |
No |
Options for Using groups to split subsets
| Option | Description | Required |
|---|---|---|
|
Splits an existing subset output into multiple files. See below. |
No |
|
Similar to |
No |
|
The ID of the subset output from |
Yes |
|
For use with Zero Input Subsetting. See examples below. |
No |
--split-by-group * outputs*
When you run smart-tests split-subset with the --split-by-group option, the CLI creates several files. If you use --output-exclusion-rules to enable Zero Input Subsetting , the behavior changes, as shown in the table below.
| File | Default | --output-exclusion-rules |
|---|---|---|
subset-groups.txt |
This file contains a list of the groups you must set up. |
This file contains a list of the groups you can skip entirely. |
subset-[groupname].txt (one file for each group) |
Each file contains the normal subset output but only for that group’s tests. You can pass these files into the test process for each group. |
Each file contains the normal subset output but only for that group’s tests. You can pass these files into the test process for each group. These files will contain exclusion rules. You’re supposed to exclude these tests. |
subset-nogroup.txt |
This file contains tests that had no group assignment, if there are any. |
This file contains tests that had no group assignment, if there are any. |
--split-by-group-with-rest * outputs*
When you run smart-tests split-subset with the --split-by-group-with-rest option, the CLI creates several files in addition to the ones described in the above table:
| File | Default | --output-exclusion-rules |
|---|---|---|
rest-groups.txt |
This file contains a list of the groups you don’t need to set up. |
This file contains a list of the groups you can’t skip. |
rest-[groupname].txt (one file for each group) |
Each file contains the normal --rest output, but only for that group’s tests. |
Each file contains the normal --rest output, but only for that group’s tests. These files will contain exclusion rules. You’re supposed to exclude these tests. |
rest-nogroup.txt |
This file contains --rest tests that had no group assignment if there are any. |
This file contains --rest tests that had no group assignment if there are any. |
Command: stats test-sessions
Retrieves statistics about test sessions
Usage: smart-tests stats test-sessions [OPTIONS]
| Option | Description | Required |
|---|---|---|
|
How many days of test sessions in the past to be stat |
No |
|
flavors (can be specified multiple times) |
No |
Output example:
{"averageDurationSeconds":653.168192926045,"count":311,"days":7}
Command: subset
Produces a subset of tests to pass to your test runner.
|
The |
Usage: smart-tests subset [OPTIONS] <TEST_RUNNER> …
| Argument | Description | Required |
|---|---|---|
|
Yes |
| Option | Description | Required |
|---|---|---|
|
(Advanced) base directory to make test names portable |
No |
|
Subsetting by confidence from 0% to 100% |
No |
|
Subsetting by programmatic goal definition |
No |
|
Ignore flaky tests above the value set by this option. You can confirm flaky scores in WebApp |
No |
|
Ignore tests that were added recently. NOTICE: this option will ignore tests that you added just now as well |
No |
|
Get subset list from guessed tests |
No |
|
Get subset list from previous full tests |
No |
|
Outputs the exclude test list. Switch the subset and rest. |
No |
|
Do not guess the base path to relativize the test file paths. By default, if the test file paths are absolute file paths, it automatically guesses the repository root directory and relativize the paths. With this option, the command doesn’t do this guess work. If --base is specified, the absolute file paths are relativized to the specified path irrelevant to this option. Use it if the guessed base path is incorrect. |
No |
|
Prioritize tests that failed within the specified hours; maximum 720 hours (= 24 hours * 30 days) |
No |
|
Prioritize tests based on test mapping file |
No |
|
Output the subset remainder to a file, e.g. --rest=remainder.txt |
No |
|
Session ID obtained by calling 'smart-tests record session'. It also accepts '@path/to/file' if the session ID is stored in a file |
Yes |
|
Subsetting target from 0% to 100% |
No |
|
Subsetting by absolute time, in seconds e.g) 300, 5m |
No |
Exactly how this command generates the subset and what’s required to do this depends on test runners. For available supported TESTRUNNER s, see Integrations .
When none of --target , --time , and --confidence is specified, CloudBees Smart Tests chooses the subset target. This is convenient on the initial setup when you are unsure what the subset size should be. Later, you can choose the right target after you see the statistics of your test suite and potential time-savings based on the CloudBees Smart Tests accumulated data.
Command: verify
Verify that the CLI can communicate with the CloudBees Smart Tests service and that you’re authenticated properly.
Usage: smart-tests verify
To avoid disrupting your CI/test process, the CloudBees Smart Tests CLI is designed to tolerate and recover from service disruptions and other recoverable error conditions by falling back to no-op. This is an intentional design, but the downside is that such transparent failures can make troubleshooting difficult.
Therefore, we recommend you keep smart-tests verify || true in a recognizable spot in your CI process. This way, when you suspect a problem in CloudBees Smart Tests, you can check the output of this command as a starting point.
Global options
|
These global options can be placed after the subcommand, for example |
--dry-run
The dry-run mode does not actually send a payload to the server, and it is helpful to check the behavior. You can also see which APIs will be requested and their payload contents in the output.
The payload contents will be output as an audit log, so if the log level is higher than the audit level, it will be forced to be set to the audit level.
Strictly speaking, it does not mean no requests will be sent, but GET requests with no payload data or side effects may be sent. This is because sometimes the response data from the GET request is needed to assemble subsequent requests.
--log-level
You can use the --log-level option to output extra information from each command.
--log-level audit is particularly useful if you want to see exactly what data gets passed to CloudBees Smart Tests when you run CLI commands. For example:
% smart-tests record build --log-level audit --build 1234 --source src=. Processed 1 commits AUDIT:smart-tests:send request method:post path:/intake/organizations/example/workspaces/awilkes/builds headers:{'Content-Type': 'application/json'} args:{'data': b'{"buildNumber": "1234", "commitHashes": [{"repositoryName": "src", "commitHash": "45b2e6d9df8e0013334354f30df1978c8b4196f8"}]}', 'timeout': (5, 60)}
--plugins
You can use the --plugins option to tell the CLI where to look for custom profiles/plugins.
For example, if you have developed (or been provided) a custom profile file named myProfile.py , place that file in the directory of your choosing (e.g., ~/smart-tests-plugins ) and use it like this:
smart-tests --plugins ~/smart-tests-plugins record tests myProfile --build $BUILD --session $SESSION /path/to/reports
Since --plugins is a global option, make sure to place it right after smart-tests but before subset or record in your command.
--skip-cert-verification
This option instructs the CLI to bypass the SSL certificate verification. This is inteded to be an escape hatch in case the system’s Python setup is broken/incomplete.
Alternatively, you can set the SMART_TESTS_SKIP_CERT_VERIFICATION environment variable to any value to have the same effect. This is more convenient if you want this behaviour across the board, instead of just one invocation.
This flag will make your communication with CloudBees Smart Tests less secure (for example, you can be vulnerable to a DNS spoofing attack). Use it with a caution.