file profile for unsupported test runners

2 minute read

This is a reference page. See Getting Started , Sending data to CloudBees Smart Tests , and Predictive Test Selection for more comprehensive usage guidelines.

About

The "file based" test runner integration is primarily designed to work with test runners that are not explicitly supported, such as custom test runners built in-house.

In order to work with CloudBees Smart Tests through this integration mechanism, your test runner has to satisfy the following conditions:

  1. File based test runner : your test runner must accept file names as an input of a test execution in order to execute just those specified set of tests.

  2. JUnit XML reports include file names/paths : your test runner has to produce results of tests in a JUnit compatible format with additional attributes that capture the file names/paths of the tests that run. If not, see Converting test reports to JUnit format .

For example, Mocha is a test runner that meets those criteria. You write tests in JavaScript files:

$ cat foo.js
var assert = require('assert');
describe('Array', function() {
  describe('#indexOf()', function() {
    it('should return -1 when the value is not present', function() {
      assert.equal([1, 2, 3].indexOf(4), -1);
    });
  });
});
▼

The Mocha test runner takes those files as arguments:

$ mocha --reporter mocha-junit-reporter foo.js

…​and produces JUnit report files, where the name of the test file is captured, in this case, in the file attribute:

$ cat test-results.xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="0.0000" tests="1" failures="0">
  <testsuite name="#indexOf()" file="/home/kohsuke/ws/foo.js" ...>
    <testcase  ... />
...
▼

The rest of this document uses Mocha as an example.

Recording test results

After running tests, point the CLI to your test report files to collect test results and train the model:

smart-tests record tests file --build <BUILD NAME> --session <SESSION_NAME> ./reports/*.xml

You might need to take extra steps to make sure that smart-tests record tests

always runs even if the build fails. See Ensuring record tests always runs .

Subsetting your test runs

The high level flow for subsetting is:

  1. Get the full list of test files and pass that to smart-tests subset with an optimization target for the subset

  2. smart-tests subset will get a subset from the CloudBees Smart Tests platform and output that list to a text file

  3. Pass the text file into your test runner to run only those tests

To retrieve a subset of tests, first pass the full list of test candidates to smart-tests subset . For example:

find ./test -name '*.js' |
smart-tests subset file \
    --build <BUILD NAME> \
    --session <SESSION_NAME> \
    --target 10% \
    --rest smart-tests-remainder.txt \
    > subset.txt
▼

  • The --build should use the same <BUILD NAME> value that you used before in smart-tests record build .

  • The --confidence option should be a percentage; we suggest 90% to start. You can also use --time or --target ; see Subsetting your test runs for more info.

This creates a file called smart-tests-subset.txt that you can pass into your command to run tests:

mocha $(< smart-tests-subset.txt)

dotnet test
Go Test