Test suite

4 minute read

If you are running different test suites against the same build, separating them into different "test suites" would improve the quality of data analysis throughout our system.

What is a test suite?

A test suite is a logical grouping of test cases within a workspace. You can organize test cases based on different criteria, such as:

  • Test type: Unit tests versus UI tests.

  • Technology stack: Maven versus Cypress.

  • Execution frequency: Tests that run on every change versus those that run nightly.

Separate test suites can run against the same build and be analyzed independently, while still contributing to the overall workspace analysis. Separate suites that run against the same build can be analyzed on their own while still contributing to the workspace as a whole.

A few habits keep suites manageable:

  • Use stable, descriptive names such as "unit", "integration", or "ui-chrome". Avoid renaming a suite once data collection is started, since other views and scripts may refer by its name.

  • Name suites by what they test or how they run, not by who runs them, so the same suite means the same thing across repositories and pipelines.

  • Keep suite names consistent across all repositories that contribute data to the workspace.

  • Each suite gets its own insights and confidence curves in the dashboard. However, model training still uses every run in the workspace.

How do you record test suite?

When you invoke the smart-tests record session command, specify the additional --test-suite option and give it a test suite name. For example, if you have a test suite called "ui tests", you would run the following command:

$ smart-tests record session --test-suite "ui tests" ...
▼

If using the smart-tests record session command explicitly as per managing complex test session layout, then the --test-suite option should be used with that command instead of record tests.

When you record test suites:

  • Teams can focus on test sessions and issues from a particular test suite.

Common ways to divide suites

The following table illustrates how test suites can be organized. The suite names are fictional examples. The names below are fictional examples.

Table 1. Common patterns
Pattern Workspace layout Test suites

One product, several repositories

One workspace, "Storefront", fed by the web, API, and mobile repositories

unit, integration, UI

Several teams, one codebase each

One workspace per team, "Team Checkout" and "Team Search"

Test suites based on the tests each team runs.

Platform with an isolated component

One workspace for the platform, a separate workspace for the component that needs its own insights.

Per workspace

Mixed runners in one repository

One workspace

One suite per runner, for example "junit" and "cypress"

Suites divided by test types and/or test runners

Many teams already divide their tests into logical groups based on:

  • The type of test (for example, unit tests vs. UI tests, etc.),

  • The tech stack used to run those tests (for example, Maven for unit tests vs. Cypress for UI tests, etc.), or

  • Both (in most cases)

If your team thinks about test suites this way, the decision should be straightforward: for example, you should send all your Maven unit tests into one workspace and all your Cypress UI tests into another.

Suites divided by test characteristics

If your team does not think about test suites this way - for example, perhaps you use a custom test runner that abstracts away some of the tech stack differences - then you should divide your tests into different workspaces. All the test info you send to a specific workspace should exhibit similar characteristics, such as:

  • Test count and test duration - For example, unit test suites typically contain a lot of many short tests, whereas UI test suites tend to have fewer tests that take longer to run.

  • Overall duration - Some test suites take minutes, whereas others can take hours of machine time to run.

  • Frequency: Many teams run different suites in different phases of their software delivery lifecycle, usually based on how long they take to run.

  • Failures: Some test suites fail more often than others; within those failures, sometimes lots of tests fail versus only a few. Similarly, some suites are more flaky than others.

Common characteristics between tests in a workspace are important for two reasons:

  • Test insights are aggregated at the workspace level. If you mix tests with different characteristics, insights such as flakiness scoring will be less useful.

  • Predictive Test Selection models are evaluated at the workspace level. If you mix tests with many different characteristics, choosing the correct optimization target will be harder.

Sub-suites within larger test suites

Sometimes teams consider small groups of tests within larger suites as suites also.

For example, a team might have a large group of tests called a "Regression test suite." Then, within that larger group, they divide tests into sub-suites based on components, like "Authentication," "API," etc. You might call these "sub-suites."

Sometimes the entire test suite runs (perhaps nightly), and other times perhaps only sub-suites are run.

If this applies to your team, we recommend using a single workspace for the entire larger suite. In the above example, that team would have one workspace for the "Regression test suite."

If you need to create another workspace in your organization, contact your customer success manager or email CloudBees Support.