CLI reference

11 minute read

Getting started

Requirements

  • Python 3.13 or newer (not required when using uv)

  • Java 8 or newer

Install

The CloudBees Smart Tests CLI is a Python3 package that you can install from PyPI.

We recommend using uv, a fast Python package installer, for the best installation experience.

First, install uv:

curl -LsSf https://astral.sh/uv/install.sh | sh export PATH="$HOME/.local/bin:$PATH"

For other installation methods (including Windows, Github Actions, etc.), see the uv installation guide.

Then install CloudBees Smart Tests CLI:

uv tool install smart-tests

Alternative: Using pip

You can also install the CLI using pip:

pip3 install --user --upgrade smart-tests

This creates a ~/.local/bin/smart-tests executable that should be in your PATH . (See PEP-370 for further details.)

Authenticate

Set your API key:

export SMART_TESTS_TOKEN=your_API_key

Verify

Then run smart-tests verify in your CI environment to see if you’ve successfully configured the CLI. If it succeeds, you’ll see a message like the one below. If you see an error message, see Troubleshooting .

$ smart-tests verify Organization: <organization> Workspace: <workspace> Proxy: None Platform: 'macOS-12.0.1-x86_64-i386-64bit' Python version: '3.9.9' Java command: 'java' smart-tests version: '1.34.0' Your CLI configuration is successfully verified 🎉

Required Workflow

With the smart-tests command update, you must now follow this mandatory sequence:

  1. smart-tests record build - Record the build information

  2. smart-tests record session - Create and register a test session (REQUIRED)

  3. smart-tests subset or smart-tests record tests - Use the session name registered in step 2 with the --session option

The session name specified in smart-tests record session must be used in subsequent commands (smart-tests subset and smart-tests record tests) via the --session parameter.

Command: inspect subset

Display the details of a subset request. See /docs/features/predictive-test-selection/#inspecting-subset-details[Subsetting your test runs] for more info.

Usage: smart-tests inspect subset [OPTIONS]

Option Description Required

--json

Display JSON format

No

--subset-id INT

Subset id

Yes

You can use smart-tests inspect subset to inspect the details of a specific subset, including rank and expected duration. This is useful for verifying that you passed the correct tests or test directory path(s) into smart-tests subset .

The output from smart-tests subset includes a tip to run smart-tests inspect subset :

$ smart-tests subset minitest --build 123 --session session-123 --confidence 90% test/*.rb > subset.txt < summary table > Run `smart-tests inspect subset --subset-id 26876` to view full subset details

Running that command will output a table containing a row for each test, including:

  • Rank/order

  • Test identifier

  • Whether the test was included in the subset

  • CloudBees Smart Tests’s estimated duration for the test * Tests with a duration of .001 seconds were not recognized by CloudBees Smart Tests

Note that the hierarchy level of the items in the list depends on the test runner in use.

For example, since Maven can accept a list of test classes as input, smart-tests inspect subset will output a prioritized list of test classes . Similarly, since Cypress can accept a list of test files as input, smart-tests inspect subset will output a list of prioritized test files . (And so on.)

Command: record attachment

Attach log files to test session. See Attaching log files for more details.

Usage: smart-tests record attachment [OPTIONS] <ATTACHMENTS>…​

Argument Description Required

<ATTACHMENTS>…​

Attachment files to upload

Yes

Option Description Required

--session SESSION

Session ID obtained by calling 'smart-tests record session'. It also accepts '@path/to/file' if the session ID is stored in a file

Yes

Command: record commit

Sends commit details to CloudBees Smart Tests. Records multiple commits from repo(s).

Usage: smart-tests record commit [OPTIONS]

Option Description Required

--import-git-log-output FILE

Import from the git-log output

No

--max-days DAYS

The maximum number of days to collect commits retroactively

No

--name NAME

Repository name

No

--source DIR

Repository path

No

Commit collection happens automatically as a part of record build , so normally this command need not be invoked separately. It’s only used for /docs/sending-data-to-smart-tests/using-the-smart-tests-cli/recording-builds-with-the-smart-tests-cli/recording-builds-from-multiple-repositories/#multiple-repositories-builtdeployed-separately-then-tested-together-eg-microservices[Multiple repositories built/deployed separately and then tested together (e.g., microservices)] .

--import-git-log-output option

If the --import-git-log-output option is used, it reads the specified file for the commit data instead of reading the commits from the repository specified by --source . The input file should contain the output of this Git command:

git log --pretty='format:{"commit": "%H", "parents": "%P", "authorEmail": "%ae", "authorTime": "%aI", "committerEmail": "%ce", "committerTime": "%cI"}' --numstat

Command: record build

Creates a record of a Build in CloudBees Smart Tests.

Usage: smart-tests record build [OPTIONS]

Option Description Required

--branch NAME

Set branch name. A branch is a set of test sessions grouped and this option value will be used for a branch name.

No

--build NAME

Build name

Yes

--commit REPO_NAME=COMMIT_HASH

Set repository name and commit hash when you use --no-commit-collection option (can be specified multiple times)

No

--link PARSE_KEY_VALUE

Set external link of a title and url (can be specified multiple times)

No

--max-days DAYS

The maximum number of days to collect commits retroactively

No

--no-commit-collection

Do not collect commit data. This is useful if the repository is a shallow clone and the RevWalk is not possible. The commit data must be collected with a separate fully-cloned repository.

No

--no-submodules

Stop collecting information from Git Submodules

No

--repo-branch-map REPO_NAME=BRANCH_NAME

Set repository name and branch name when you use --no-commit-collection option. Please use the same repository name with a commit option (can be specified multiple times)

No

--source DIR

Path to local Git workspace, optionally prefixed by a label. like --source path/to/ws or --source main=path/to/ws (can be specified multiple times)

No

--timestamp TIMESTAMP

Used to overwrite the build time when importing historical data. Note: Format must be YYYY-MM-DDThh:mm:ssTZD or YYYY-MM-DDThh:mm:ss (local timezone applied)

No

The act of recording a build teaches CloudBees Smart Tests that the specified set of commits has turned into a build and that the given name henceforth identifies this build. This forms the basis of how CloudBees Smart Tests calculates the changes.

Conceptually, a build is a collection of Git repositories, each at a specific commit. REPO_NAME identifies each repository contributing to a build, and it needs to be stable across different builds of the same project. Good examples include:

  • Relative directory paths to the repository from the "workspace root," such as src/moduleX if they are stable.

  • GitHubOrg/GitHubRepo slug if your repositories are on GitHub since they are also stable.

Command: record session

This command is now mandatory and must be executed after smart-tests record build and before smart-tests subset or smart-tests record tests.

Creates a record of a test session in CloudBees Smart Tests.

Usage: smart-tests record session [OPTIONS]

Option Description Required

--build NAME

Build name

Yes

--flavor KEY=VALUE

Flavors (can be specified multiple times)

No

--no-build

If you want to only send test reports, please use this option

No

--observation

Enable observation mode

No

--link TITLE=URL

Set external link of title and url (can be specified multiple times)

No

--test-suite NAME

Set test suite name. A test suite is a collection of test sessions. Setting a test suite allows you to manage data over test sessions and lineages.

Yes

--timestamp TIMESTAMP

Used to overwrite the session time when importing historical data. Note: Format must be YYYY-MM-DDThh:mm:ssTZD or YYYY-MM-DDThh:mm:ss (local timezone applied)

No

This command tells CloudBees Smart Tests that you are about to begin testing a build that was been recorded earlier with the record build command. This command is now mandatory and must be executed after every smart-tests record build command.

The command outputs a string you can save for use in other commands (like smart-tests subset and smart-tests record tests) instead of --build . We suggest saving the value either to an environment variable or to a text file:

# environment variable smart-tests record build --build BUILD_NAME [OPTIONS] export SMART_TESTS_SESSION=$(smart-tests record session --build BUILD_NAME --session SESSION_NAME [OPTIONS]) <run tests> smart-tests record tests TESTRUNNER --session SESSION_NAME [OPTIONS]
# text file smart-tests record build --build BUILD_NAME [OPTIONS] smart-tests record session --build BUILD_NAME --session SESSION_NAME [OPTIONS] <run tests> smart-tests record tests TESTRUNNER --session SESSION_NAME [OPTIONS]

(Otherwise, the command will write a session ID to ~/.config/smart-tests/sessions/{hash}.txt . This location may change in the future, so don’t rely on it.)

Command: record tests

Send test results for the test session to CloudBees Smart Tests.

The --session parameter is required and must use the session name that was registered with smart-tests record session.

Usage: smart-tests record tests [OPTIONS] <TEST_RUNNER> …​

Argument Description Required

<TEST_RUNNER>

Yes

Option Description Required

--base CONVERT

(Advanced) base directory to make test names portable

No

--group NAME

Grouping name for test results

No

--no-base-path-inference

Do not guess the base path to relativize the test file paths. By default, if the test file paths are absolute file paths, it automatically guesses the repository root directory and relativize the paths. With this option, the command doesn’t do this guess work. If --base-path is specified, the absolute file paths are relativized to the specified path irrelevant to this option. Use it if the guessed base path is incorrect.

No

--post-chunk INT

Post chunk

No

--session SESSION

Session ID obtained by calling 'smart-tests record session'. It also accepts '@path/to/file' if the session ID is stored in a file

Yes

This command reads JUnit (or similar) XML report files produced by test runners and sends them to CloudBees Smart Tests.

Exactly how this command generates the subset and what’s required to do this depends on test runners. For available supported TESTRUNNER , see Integrations

Command: split-subset

Splits an existing subset from CloudBees Smart Tests into chunks. This relates to Replacing static parallel suites with a dynamic parallel subset and Using groups to split subsets .

smart-tests split-subset TESTRUNNER [OPTIONS] …​

Intended for use with smart-tests subset with the --split option.

Options for Replacing static parallel suites with a dynamic parallel subset

Option Description Required

--bin BIN_NUMBER/BIN_COUNT

The portion of the subset to retrieve, e.g --bin 1/3

Yes

--rest FILE

Output the remainder of the subset to a file. This is useful for running the "rest of the tests" after you’ve run a subset.

No

--same-bin FILE

[Beta; Gradle only] Place tests listed in the FILE to belong to the same bin to avoid the tests running simultaneously.

No

--subset-id SUBSET-ID-STRING

The ID of the subset output from smart-tests subset --split …​ (see --split under subset)

Yes

--output-exclusion-rules

Output a list of tests to exclude instead of a list to include. See Zero Input Subsetting.

No

Options for Using groups to split subsets

Option Description Required

--split-by-group

Splits an existing subset output into multiple files. See below.

No

--split-by-group-with-rest

Similar to --split-by-group, except remainder/rest files are also included. See below.

No

--subset-id SUBSETID

The ID of the subset output from smart-tests subset --split …​ (see --split under subset)

Yes

--output-exclusion-rules

For use with Zero Input Subsetting. See examples below.

No

--split-by-group * outputs*

When you run smart-tests split-subset with the --split-by-group option, the CLI creates several files. If you use --output-exclusion-rules to enable Zero Input Subsetting , the behavior changes, as shown in the table below.

File Default --output-exclusion-rules

subset-groups.txt

This file contains a list of the groups you must set up.

This file contains a list of the groups you can skip entirely.

subset-[groupname].txt (one file for each group)

Each file contains the normal subset output but only for that group’s tests. You can pass these files into the test process for each group.

Each file contains the normal subset output but only for that group’s tests. You can pass these files into the test process for each group. These files will contain exclusion rules. You’re supposed to exclude these tests.

subset-nogroup.txt

This file contains tests that had no group assignment, if there are any.

This file contains tests that had no group assignment, if there are any.

--split-by-group-with-rest * outputs*

When you run smart-tests split-subset with the --split-by-group-with-rest option, the CLI creates several files in addition to the ones described in the above table:

File Default --output-exclusion-rules

rest-groups.txt

This file contains a list of the groups you don’t need to set up.

This file contains a list of the groups you can’t skip.

rest-[groupname].txt (one file for each group)

Each file contains the normal --rest output, but only for that group’s tests.

Each file contains the normal --rest output, but only for that group’s tests. These files will contain exclusion rules. You’re supposed to exclude these tests.

rest-nogroup.txt

This file contains --rest tests that had no group assignment if there are any.

This file contains --rest tests that had no group assignment if there are any.

Command: stats test-sessions

Retrieves statistics about test sessions

Usage: smart-tests stats test-sessions [OPTIONS]

Option Description Required

--days INT

How many days of test sessions in the past to be stat

No

--flavor KEY=VALUE

flavors (can be specified multiple times)

No

Output example:

{"averageDurationSeconds":653.168192926045,"count":311,"days":7}

Command: subset

Produces a subset of tests to pass to your test runner.

The --session parameter is required and must use the session name that was registered with smart-tests record session.

Usage: smart-tests subset [OPTIONS] <TEST_RUNNER> …​

Argument Description Required

<TEST_RUNNER>

Yes

Option Description Required

--base DIR

(Advanced) base directory to make test names portable

No

--confidence PERCENTAGE

Subsetting by confidence from 0% to 100%

No

--goal-spec GOAL_SPEC

Subsetting by programmatic goal definition

No

--ignore-flaky-tests-above N

Ignore flaky tests above the value set by this option. You can confirm flaky scores in WebApp

No

--ignore-new-tests

Ignore tests that were added recently. NOTICE: this option will ignore tests that you added just now as well

No

--get-tests-from-guess

Get subset list from guessed tests

No

--get-tests-from-previous-sessions

Get subset list from previous full tests

No

--output-exclusion-rules

Outputs the exclude test list. Switch the subset and rest.

No

--no-base-path-inference

Do not guess the base path to relativize the test file paths. By default, if the test file paths are absolute file paths, it automatically guesses the repository root directory and relativize the paths. With this option, the command doesn’t do this guess work. If --base is specified, the absolute file paths are relativized to the specified path irrelevant to this option. Use it if the guessed base path is incorrect.

No

--prioritize-tests-failed-within-hours N

Prioritize tests that failed within the specified hours; maximum 720 hours (= 24 hours * 30 days)

No

--prioritized-tests-mapping FILE

Prioritize tests based on test mapping file

No

--rest FILE

Output the subset remainder to a file, e.g. --rest=remainder.txt

No

--session SESSION

Session ID obtained by calling 'smart-tests record session'. It also accepts '@path/to/file' if the session ID is stored in a file

Yes

--target PERCENTAGE

Subsetting target from 0% to 100%

No

--time TIME

Subsetting by absolute time, in seconds e.g) 300, 5m

No

Exactly how this command generates the subset and what’s required to do this depends on test runners. For available supported TESTRUNNER s, see Integrations .

When none of --target , --time , and --confidence is specified, CloudBees Smart Tests chooses the subset target. This is convenient on the initial setup when you are unsure what the subset size should be. Later, you can choose the right target after you see the statistics of your test suite and potential time-savings based on the CloudBees Smart Tests accumulated data.

Command: verify

Verify that the CLI can communicate with the CloudBees Smart Tests service and that you’re authenticated properly.

Usage: smart-tests verify

To avoid disrupting your CI/test process, the CloudBees Smart Tests CLI is designed to tolerate and recover from service disruptions and other recoverable error conditions by falling back to no-op. This is an intentional design, but the downside is that such transparent failures can make troubleshooting difficult.

Therefore, we recommend you keep smart-tests verify || true in a recognizable spot in your CI process. This way, when you suspect a problem in CloudBees Smart Tests, you can check the output of this command as a starting point.

Global options

These global options can be placed after the subcommand, for example smart-tests verify --log-level audit

--dry-run

The dry-run mode does not actually send a payload to the server, and it is helpful to check the behavior. You can also see which APIs will be requested and their payload contents in the output.

The payload contents will be output as an audit log, so if the log level is higher than the audit level, it will be forced to be set to the audit level.

Strictly speaking, it does not mean no requests will be sent, but GET requests with no payload data or side effects may be sent. This is because sometimes the response data from the GET request is needed to assemble subsequent requests.

--log-level

You can use the --log-level option to output extra information from each command.

--log-level audit is particularly useful if you want to see exactly what data gets passed to CloudBees Smart Tests when you run CLI commands. For example:

% smart-tests record build --log-level audit --build 1234 --source src=. Processed 1 commits AUDIT:smart-tests:send request method:post path:/intake/organizations/example/workspaces/awilkes/builds headers:{'Content-Type': 'application/json'} args:{'data': b'{"buildNumber": "1234", "commitHashes": [{"repositoryName": "src", "commitHash": "45b2e6d9df8e0013334354f30df1978c8b4196f8"}]}', 'timeout': (5, 60)}

--plugins

You can use the --plugins option to tell the CLI where to look for custom profiles/plugins.

For example, if you have developed (or been provided) a custom profile file named myProfile.py , place that file in the directory of your choosing (e.g., ~/smart-tests-plugins ) and use it like this:

smart-tests --plugins ~/smart-tests-plugins record tests myProfile --build $BUILD --session $SESSION /path/to/reports

Since --plugins is a global option, make sure to place it right after smart-tests but before subset or record in your command.

--skip-cert-verification

This option instructs the CLI to bypass the SSL certificate verification. This is inteded to be an escape hatch in case the system’s Python setup is broken/incomplete.

Alternatively, you can set the SMART_TESTS_SKIP_CERT_VERIFICATION environment variable to any value to have the same effect. This is more convenient if you want this behaviour across the board, instead of just one invocation.

This flag will make your communication with CloudBees Smart Tests less secure (for example, you can be vulnerable to a DNS spoofing attack). Use it with a caution.