Introduction
IntroductionUse the CLI with a public repositoryUpdate tokenless authentication to use GitHub OIDCUse the CLI with Docker image
IntroductionChoose a value for build nameRecord builds from multiple repositoriesRecord builds for microservicesRecord builds for multistage buildRun under restricted networksDeal with shallow clones
Record sessions
IntroductionRequest and run a subset of testsSubset with the CloudBees Smart Tests CLIChoose a subset optimization targetObserve subset behaviorSmart subset optimization targetsZero Input SubsettingReplace static parallel suites with a dynamic parallel subsetUse groups to split subsetsCombine with rule-based test selectionCloudBees Smart Tests extension for VS code
IntroductionConvert test reports to JUnit formatEnsure record tests always runsManage complex test session layoutsUse 'flavors' to run the best tests for an environmentSeparate out test suitesAttach log files

Troubleshoot

3 minute read

The following sections describe common issues that you might encounter when using CloudBees Smart Tests and how to resolve them.

"smart-tests verify" failure

There are a number of reasons why smart-tests verify might fail. The most common ones are connectivity issues, which can be caused by firewalls, proxies, or SSL certificate problems. The following provide guidance on how to troubleshoot these issues.

Firewalls and static IP addresses

If you receive an error like this one, then you will need to configure your firewall to allow traffic to api-static.mercury.launchableinc.com:

$ smart-tests verify
unable to post to https://api.mercury.launchableinc.com/...

$ smart-tests record build
...
Exception in thread "main" java.net.UnknownHostException: api.mercury.launchableinc.com: No address associated with hostname
▼

If you need to interact with the API via static IPs, first set the SMART_TESTS_BASE_URL environment variable to https://api-static.mercury.launchableinc.com .

The IP for this hostname will be either 13.248.185.38 or 76.223.54.162 which you can add to your firewall settings.

Proxies

If your CI server sits behind a proxy, you can tell the CLI to use it by setting the HTTP_PROXY and/or HTTPS_PROXY environment variables. For example:

export HTTP_PROXY="http://10.10.1.10:3128"
export HTTPS_PROXY="http://10.10.1.10:1080"
▼

SSL certificate verification error

If you get an error like this:

`SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate `

It is an indication that your system is lacking the root CA certificates. See the documentation for Requests, which the CLI uses under the hood, for how it looks up certificates. Also check out Stack Overflow posts like this where people discusses various remedies.

If all else fails, use the --skip-cert-verification option of the CloudBees Smart Tests CLI to altogether bypass the SSL certificate check process. This means you are susceptible to MITM attacks, so use it with caution.

Missing Git object during commit collection

During record build or record commit , you might encounter errors like this:

Warning: 5730667 is missing. Skipping diff calculation for f2244ec → 20630e5.

This warning indicates that a Git object (typically a "blob", which stores a file) is missing from the local clone, preventing the calculation of a diff between the two specified commits. This is typically caused by a Git shallow clone or Git partial clone.

CloudBees Smart Tests will continue to function without this information, hence this is a warning; Predictive test selection will run with limited set of information, which will hurt the performance by unknown amount. If you are only seeing this sporadically, we suggest you simply ignore this warning.

Getting rid of this warning involves tweaking the git-clone or git-fetch operations to ensure enough of the relevant Git objects are made available locally. For example,

  • CI systems often use --depth 1 to just fetch the very latest commit and that alone. Increase this number should help, say --depth 32.

  • GitHub Action actions/checkout uses --filter=blob:none when you choose to do a sparse checkout. Use filter: none to override this.

Recording branch of build

During record build, CloudBees Smart Tests tries to automatically determine the Git branch being built. However, this can fail, depending on how you check out the code. When this happens, our webapp will ask you to configure this explicitly.

If your build involves just one repository, you can do this by using the --branch NAME option:

smart-tests record build --build $BUILD_NAME --source . --branch $GIT_BRANCH

If you have multiple repositories, you have to use the --branch REPONAME=BRANCHNAME format:

smart-tests record build --build $BUILD_NAME \
  --source product=. --source test=../test --branch product=$GIT_BRANCH
▼

You can specify branches for every repository in this manner. However, the first repository is the most important because it is used as the branch for this build as a whole. CloudBees Smart Tests strongly recommends choosing the primary repository as the first repository for this reason.

Security policies