Replacing static parallel suites with a dynamic parallel subset

Some teams manually split their test suites into several "bins" to run them in parallel. This presents a challenge in adopting CloudBees Smart Tests because you don’t want to lose the benefit of parallelization.

With split subsets , you can replace your manually selected bins with automatically populated bins from a CloudBees Smart Tests subset.

For example, let’s say you currently run ~80 minutes of tests split coarsely into four bins and run in parallel across four workers:

Worker 1: ~20 minutes of tests
Worker 2: ~15 minutes of tests
Worker 3: ~20 minutes of tests
Worker 4: ~25 minutes of tests

With a split subset, you can generate a subset of the full 80 minutes of tests and then call CloudBees Smart Tests once in each worker to get the bin of tests for that runner.

The high-level flow is:

Request a subset of tests to run from CloudBees Smart Tests by running smart-tests subset with the --split option. Instead of outputting a list of tests, the command will output a subset ID that you should save and pass into each runner.
Start up your parallel test worker, e.g., four runners from the example above
Request the bin of tests that the worker should run. To do this, run smart-tests split-subset with: . the --subset-id option set to the ID you saved earlier, and
the --bin value set to bin-number/bin-count .
If you’re using Zero Input Subsetting , add the --output-exclusion-rules option.
Run the tests on each worker.
After each run finishes in each worker, record test results using smart-tests record tests with the --subset-id option set to the ID you saved earlier.

In pseudocode:

# main
$ smart-tests record build --build $BUILD_ID --source src=.
$ smart-tests record session --build $BUILD_ID --session $SESSION_ID
$ smart-tests subset --split --confidence 90% --build $BUILD_ID --session $SESSION_ID bazel .
subset/12345

...

# worker 1
$ smart-tests split-subset bazel --subset-id subset/12345 --bin 1/3 --rest rest.txt > subset.txt
$ bazel test $(cat subset.txt)
$ smart-tests record tests --build $BUILD_ID --session $SESSION_ID --subset-id subset/12345 bazel .


# worker 2
$ smart-tests split-subset bazel --subset-id subset/12345 --bin 2/3 --rest rest.txt > subset.txt
$ bazel test $(cat subset.txt)
$ smart-tests record tests --build $BUILD_ID --session $SESSION_ID --subset-id subset/12345 bazel .

# worker 3
$ smart-tests split-subset bazel --subset-id subset/12345 --bin 3/3 --rest rest.txt > subset.txt
$ bazel test $(cat subset.txt)
$ smart-tests record tests --build $BUILD_ID --session $SESSION_ID --subset-id subset/12345 bazel .▼

[Beta; Gradle only] Dynamic parallel subset with same bin option

Even though smart-tests split-subset offers evenly split subset bins, there are some cases where certain tests should not run concurrently. You want to use split-subset to split a subset into multiple bins and want certain tests to belong to the same bin to avoid simultaneous execution.

E.g., TestA and TestB use the same record in a database, and their concurrent access may cause the tests to fail.

We provide the --same-bin <file.txt> option to avoid running these tests simultaneously. By adding --same-bin <file.txt> option to split-subset , the test cases listed in the <file.txt> will be placed in the same bin.

In pseudocode:

# main
$ smart-tests subset --target 90% --build BUILD_ID --session $SESSION_ID --split gradle src/test/java
subset/12345
Your model is currently in training
{PRODUCT} created subset 12345 for build test (test session 12345) in workspace launchableinc/mothership

|           |   Candidates |   Estimated duration (%) |   Estimated duration (min) |
|-----------|--------------|--------------------------|----------------------------|
| Subset    |            7 |                  77.7778 |                0.000116667 |
| Remainder |            2 |                  22.2222 |                3.33333e-05 |
|           |              |                          |                            |
| Total     |            9 |                 100      |                0.00015     |

Run `smart-tests inspect subset --subset-id 12345` to view full subset details

---

# worker 1
$ cat same_bin0.txt
example.DB0Test
example.DB1Test

$ smart-tests split-subset \
    --subset-id subset/12345 \
    --bin 1/2 \
    --same-bin same_bin0.txt \
    gradle
--tests example.DB0Test --tests example.DB1Test --tests example.MulTest --tests example.DivTest

---

# worker 2
$ cat same_bin0.txt
example.DB0Test
example.DB1Test

$ smart-tests split-subset \
    --subset-id subset/12345 \
    --bin 2/2 \
    --same-bin same_bin0.txt \
    gradle
--tests example.AddTest --tests example.SubTest --tests example.PowTest▼