Managing complex test session layouts

This page relates to Recording test results with the CloudBees Smart Tests CLI.

The Test Session is one of CloudBees Smart Tests’s core concepts. When you record test results, those results are recorded to a test session. When you request a subset of tests from CloudBees Smart Tests, the subset is linked to a test session, too. This concept is useful because tests might run several times against the same build; it helps disambiguate those runs.

Normally, the CloudBees Smart Tests CLI manages session creation and usage in the background. However, if your build and test processes are split across multiple machines, or if your tests are parallelized across multiple machines, you’ll need to create test sessions yourself.

Build and test processes happen on different machines

Normally, the CloudBees Smart Tests CLI handles creating, saving, and retrieving a session ID in the background. When you run smart-tests subset or smart-tests record tests , the CLI checks for an existing file in ~/.launchable . This file is written when you run smart-tests record build .

However, if you need to record tests ( smart-tests record tests ) or request a subset ( smart-tests subset ) on a different machine than the one where smart-tests record build ran, the ~/.launchable file won’t be present, and you’ll need to manually create a test session using the smart-tests record session command at the beginning of your test process.

This command outputs a string that you can store and then pass into the --session option in smart-tests subset and smart-tests record tests .

Here’s some pseudocode to illustrate:

# machine 1

    # build step
    smart-tests record build --build <BUILD NAME> --branch <BRANCH NAME> [OPTIONS]

    # build software
    bundle install

# machine 2

    # before running tests, create a session (mandatory)
    # you'll use this session name again later
    smart-tests record session --build <BUILD NAME> --session <SESSION_NAME> [OPTIONS]

    # get a subset, if applicable
    smart-tests subset --build <BUILD NAME> --session <SESSION_NAME> [OPTIONS]

    # run tests
    bundle exec rails test [OPTIONS]

# machine 3

    # record test results
    smart-tests record tests --session <SESSION_NAME> [OPTIONS]▼

Combining test reports from multiple runs

Some pipelines execute multiple test runs against a build, outputting distinct test report(s) across several machines. Depending on your layout (see Test Session ), you may want to combine these into a single test session. This section explains how to do this.

This may also be the case if you execute tests of a single type across several parallel runs, but usually the test runner can combine reports from parallel runs for consumption from a single place.

If all the test reports for a session can be collected from a single machine, you don’t need to use this method.

To tie multiple smart-tests record tests invocations to a single test session:

Use the smart-tests record session command to create a session ID
Pass this ID into the corresponding --session parameter in smart-tests record tests (note: pseudocode) :

## build step

# before building software, send commit and build info
# to {PRODUCT}
smart-tests record build --build <BUILD NAME> [OPTIONS]

# build software the way you normally do, for example
bundle install

## test step

# before running tests, create a session (mandatory)
# you'll use this session name to group the test reports together later
smart-tests record session --build <BUILD NAME> --session <SESSION_NAME> [OPTIONS]

    # start multiple test runs

        # machine 1

            # run tests
            bundle exec rails test

            # send test results to {PRODUCT} from machine 1
            # Note: You need to configure the line to always run whether test run succeeds/fails.
            #       See each integration page.
            smart-tests record tests --build <BUILD NAME> --session <SESSION_NAME> [OPTIONS]

        # machine 2

            # run tests
            bundle exec rails test

            # send test results to {PRODUCT} from machine 2
            # Note: You need to configure the line to always run whether test run succeeds/fails.
            #       See each integration page.
            smart-tests record tests --build <BUILD NAME> --session <SESSION_NAME> [OPTIONS]

        ## repeat as needed...

## finish multiple test runs▼

You can read more about smart-tests record session in the CLI reference.