Capturing test reports and selecting tests to run in multiple environments (e.g. browser tests, mobile tests, etc.)
|
This page relates to Recording test results with the CloudBees Smart Tests CLI. |
Lots of teams run the same tests across several different environments. For example, a UI test suite might be run in several browsers in parallel. Or perhaps you need to build a slightly different version of a mobile app for different locales and need to run the same tests across all of them.
In these scenarios, a test result is not just a test result: it is the combination of the test and the environment that it was run in. A test might pass in one environment but fail in another.
CloudBees Smart Tests supports these scenarios with a concept called flavors .
When you submit test results using smart-tests record tests , you can submit additional metadata in the form of key-value pairs using the --flavor option.
For example:
# run tests in Chrome and report results cypress run --reporter junit --reporter-options "mochaFile=report/test-output-chrome.xml" smart-tests record tests cypress --build [BUILD NAME] --session [SESSION NAME] --flavor browser=chrome report/test-output-chrome.xml # run tests in Firefox and report results cypress run --reporter junit --reporter-options "mochaFile=report/test-output-firefox.xml" smart-tests record tests cypress --build [BUILD NAME] --session [SESSION NAME] --flavor browser=firefox report/test-output-firefox.xml
And so on. (You can submit multiple key-value pairs, too: --flavor key=value --flavor key2=value2 )
Later, when you want to request a subset of tests, you can include the same key-value pairs to get a subset of tests specifically selected for that flavor.
For example:
# get a subset for Chrome, run it, then report results find ./cypress/integration | smart-tests subset cypress --build [BUILD NAME] --session [SESSION NAME] --confidence 90% --flavor browser=chrome > subset-chrome.txt cypress run --spec "$(cat subset-chrome.txt)" --reporter junit --reporter-options "mochaFile=report/test-output-chrome.xml" smart-tests record tests cypress --build [BUILD NAME] --session [SESSION NAME] --flavor browser=chrome report/test-output-chrome.xml # get a subset for Firefox, run it, then report results find ./cypress/integration | smart-tests subset --build [BUILD NAME] --session [SESSION NAME] --confidence 90% --flavor browser=firefox cypress > subset-firefox.txt cypress run --spec "$(cat subset-firefox.txt)" --reporter junit --reporter-options "mochaFile=report/test-output-firefox.xml" smart-tests record tests cypress --build [BUILD NAME] --session [SESSION NAME] --flavor browser=firefox report/test-output-firefox.xml
This feature lets you select the right tests to run based on the changes being tested and the environment they are being run in.
Note: if your workflow involves creating a session externally using smart-tests record session , you should set --flavor in that command (instead of smart-tests subset or smart-tests record tests , as they will be ignored), such as:
smart-tests record session --build [BUILD NAME] --session chrome-session --flavor browser=chrome find ./cypress/integration | smart-tests subset cypress --session chrome-session --confidence 90% > subset-chrome.txt cypress run --spec "$(cat subset-chrome.txt)" --reporter junit --reporter-options "mochaFile=report/test-output-chrome.xml" smart-tests record tests cypress --session chrome-session report/test-output-chrome.xml