Using groups to split subsets

It’s common for teams to split up a large test suite into many smaller groups, often executed in parallel.

Perhaps your pipeline tests multiple components or plugins, or maybe your framework forces this kind of organization (e.g., dotnet test organizes tests by .dll ). In any case, all the groups are tested together (comprising a single Test Session ), but each group has its own small test suite.

To better support this scenario, Zero Input Subsetting makes it possible to request a single "high level" subset across all components.

Now, a new concept called groups intends to improve the usability of this approach:

First, you can now use --split-by-group to split the "high level" subset file into one file for each group, simplifying test distribution across groups.
Second, when you use --split-by-group , the CLI writes a special file informing you which groups you can skip entirely, saving setup time

Currently, only the Maven profile supports test groups, so this document uses instructions for Maven.

Assigning tests to groups

Before you can use --split-by-group , you need to assign your tests to their respective groups. Each test can belong to one test group at a time. A group aligns with component/plugin/DLL — whatever your organizational container is.

You can assign a set of tests to a group by running smart-tests record tests with the --group=[groupName] option.

This means you’ll run smart-tests record tests ten times if you have ten groups.

Example: Group assignment using the CLI

For example, we have three components: A, B, and C. Each group has 5 test items. We’ll assign each to groups. (Note the use of --group on each smart-tests record tests )

For clarity, here are the commands:

# before building software
smart-tests record build \
    --name jenkins-build-123\
    [...other options]

...[build steps]...

# before running tests create a test session so we can collect all the results together
smart-tests record session \
    --build jenkins-build-123 \
    --session test-session-123

    # componentA tests
    ...[run componentA tests]...
    smart-tests record tests \
        --build jenkins-build-123 \
        --session test-session-123 \
        --group=componentA \
        [...other options] \
        /path/to/componentA/results

    # componentB tests
    ...[run componentB tests]...
    smart-tests record tests \
        --build jenkins-build-123 \
        --session test-session-123 \
        --group=componentB \
        [...other options] \
        /path/to/componentB/results

    # componentB tests
    ...[run componentC tests]...
    smart-tests record tests \
        --build jenkins-build-123 \
        --session test-session-123 \
        --group=componentC \
        [...other options] \
        /path/to/componentC/results▼

The examples on this page describe a scenario with only 3 groups. This is just for illustrative purposes. In reality, this approach is for teams with lots of groups (e.g. 10+).

Splitting subsets by group

Once you’ve assigned your tests to groups, you can create a high-level subset and split it by group. This involves two commands run one after another in your main CI pipeline before you run any component pipelines:

smart-tests subset with the --split option added. This option modifies the command’s output to return a subset ID string instead of the subset contents. You’ll use this ID in the next command. * Include the --get-tests-from-previous-sessions and --output-exclusion-rules options.
- CloudBees Smart Tests creates a "high-level" exclusion list in this step and stores it for retrieval in the next step.
smart-tests split-subset with the --split-by-group option. This command outputs several files for your pipeline (see below). * The --subset-id option is also required. This uses the value from the previous command.
- In this step, CloudBees Smart Tests splits the just-created exclusion list by group.

Special output files

When you run smart-tests split-subset with the --split-by-group and the --output-exclusion-rules option, the CLI creates several files:

subset-groups.txt * Since you used --output-exclusion-rules with smart-tests subset , this file contains a list of the groups you can skip entirely.
subset-[groupname].txt (one file for each group) * Each file contains the normal subset output but only for that group’s tests. You can pass these files into the test process for each group.
Since you used --output-exclusion-rules with smart-tests subset , these files contain exclusion rules. You’re supposed to exclude these tests.
subset-nogroup.txt * This file contains tests that had no group assignment, if there are any.

See the CLI reference for additional options.

Example: Split output by group using the CLI

In this example, we’ll continue the scenario from above. We have three groups, each with five tests. We’ve already assigned each test to its respective group. Now we want to use that.

This diagram shows the flow. First, we create a subset from all the tests across all groups. Then we split those into groups. Note the special file subset-groups.txt , which shows us we can skip component B entirely.

Note that the diagram shows the contents of subset-component*.txt as a list of classes. This is the output format for Maven.

If you use a different test runner, your output might be different. Every test runner has its own exclusion syntax.

# before building software
smart-tests record build \
    --name jenkins-build-123\
    [...other options]

...[build steps]...

# before running tests create a test session so we can collect all the results together
smart-tests record session \
    --build jenkins-build-123 \
    --session test-session-123

# create the server side subset
smart-tests subset \
    --build jenkins-build-123 \
    --session test-session-123 \
    --split \
    --get-tests-from-previous-sessions \
    --output-exclusion-rules > subset-id.txt

# split that for use locally
smart-tests split-subset \
    --subset-id $(cat subset-id.txt) \
    --split-by-groups \
    --output-exclusion-rules \
    maven▼

At this stage, we have several files:

subset-groups.txt * componentB
subset-componentA.txt * Exclusion rules for component A
subset-componentB.txt * Exclusion rules for component B
subset-componentC.txt * Exclusion rules for component C

Because subset-groups.txt contains componentB , we can write a script to skip that group’s test setup entirely. (How you do this depends on your setup. Need help? Let us know.)

Finally, we pass each component’s exclusion file into the remaining test processes for each group:

subset-componentA.txt gets passed into the test process for component A
subset-componentC.txt gets passed into the test process for component C

We follow the normal instructions for using an exclusion rule (see the documentation for your test runner) so that only those tests run.

For example, here’s a basic invocation of Maven for component A, complete with test recording at the end:

# component A
# run component A tests with {PRODUCT} exclusions
mvn test -Dsurefire.excludesFile=$PWD/subset-componentA.txt

# record tests
smart-tests record tests \
        --session test-session-123 \
        --group=componentA \
        [...other options] \
        /path/to/componentA/results▼

As a result, we entirely eliminated component B (in this example) from the test process, saving time!

A note about new tests

Because component B was entirely skipped, if there are any new group B tests, they didn’t get run. This is the tradeoff of saving the setup time. To mitigate this, you should have a full run scheduled later in your pipeline, so they’ll run then.