Automated testing is critical to identifying errors early in your pipeline and ensuring quality software is delivered to production. The results from automated tests can determine whether a pipeline should be promoted to the next stage, or if the release should be stopped from moving forward. Driving test automation flows at the right time in the pipeline becomes even more critical in order to meet the business desires of releasing more frequently.
CloudBees Flow integrates with many testing tools and frameworks to run automated tests as part of the delivery pipeline. The results of these tests can be linked into the pipeline summary to have a central view of all of the data associated with the release. Test logs can be parsed to extract results and specific metrics or data, and then this data can be bubbled up for visibility and to control promotion to the next stage using automated gates.
Acting on Test Results
Your pipeline should be data driven and should respond based on the outcome of testing with a "go or no go" on continuing the release. In line with the goal of reducing cycle times and improving quality and efficiency in your releases, these checks of test results should be an automated part of your pipeline rather than requiring a manual check of the results. After running the tests and gathering the results, the pipeline can have policy-based approvals to ensure that the test results meet a certain threshold before promoting further. Use automated gates and run conditions in CloudBees Flow pipelines to check test outcomes before continuing.
When going from Dev to Staging, an automated check of test results may be a sufficient gate between the stages. As you move further right in the pipeline, it may require a combination of automated checks with manual approvals before the pipeline can be promoted to controlled environments.
Collecting and Parsing Test Data
CloudBees Flow stores the standard output from any commands in log files, which can easily be drilled into to see the details of what was run and troubleshoot any issue. When running large test suites, it is not always easy to scan a long log file and find the data you need. You may need to extract certain results or numbers for easy visibility and for use later in the pipeline. In fact, many times you need to understand the log as it is generated in real time and take corrective action immediately instead of waiting for the entire process to finish.
The CloudBees Flow PostP (or PostProcessor) feature is a powerful and flexible way to extract important details and values from the job’s log file in real time. With customizable pattern matching, PostP parses the log file as the job is running to find interesting test data that matches your regular expressions. This data can then be stored in properties that can be retrieved and acted on later in the pipeline. There are a variety of other actions that can be taken, such as extracting sections of the log file or counting the number of occurrences of a pattern. For example, PostP can be used to fail a pipeline fast if you realize that first few critical test cases of a normally long-running functional test have failed and that there is no need to continue the tests.
See Postprocessors: Collecting Data for Reports for more detail on how to write your custom matchers as well as the functions available for parsing and extracting data with PostP.
Example: Test Automation Driving the CloudBees Flow Pipeline
This example illustrates how you can aggregate your automated testing data, extract the details, and act on results to drive your pipeline.
The following figure shows the pipeline discussed in this example:
The pipeline has three stages, DEV, QA, and PROD. The DEV and QA stages run the application deployment process and then start a suite of Selenium Tests, calling the EC-Selenium plugin to invoke the tests. You can replace the testing task with any testing tool by using one of the test plugins available in CloudBees Flow.
A link to the test results is stored in the pipeline summary. Anyone viewing or approving the pipeline has direct access to the full details of the tests that were run for the environment. All data is aggregated centrally. For details on adding links to a stage summary, see Linking to Stage Summary Information .
The Selenium test steps leverage PostP to extract the data on successful tests passed and store them in a property in the pipeline stage named seleniumTestsPassed. These properties are then used in the automated gates between pipeline stages.
For details on defining automated gates and adding conditions, see Pipeline Concepts and Creating a Condition-Based Approval Rule in a Gate .
QA to PROD Promotion
There is one gate between the QA and PROD stages. This gate has a combination of automation and manual intervention. Before entering the gate, there is an automated condition that checks the percentage of Selenium tests that passed. If the percent is greater than 90, the manual check is skipped, and the pipeline starts the PROD stage. If less than 90% of tests passed, the manual gate is invoked, and the QA engineer must view the test results to see whether the application can be promoted to PROD.
Condition on the gate:
The manual gate is triggered only if the condition is met:
QA to Production Promotion
The promotion from QA to Production requires more diligence than the promotion to QA. Here there are two gates, one automated and a second manual approval, required for promotion. The first automated gate, as with the Dev stage, is based on the test results but requires that 100% of the tests passed in order to continue. If the automated gate is rejected, then the pipeline is aborted. If the automated gate is approved, then a manual gate is triggered, where the release manager will view the pipeline details with quick access to the testing results linked to the stage summary and then manually approve or reject the gate.
Running the Pipeline
Following is a walk-through of a pipeline run.
In the Dev stage below, 95% of Selenium tests passed. The test results are shown and linked in the stage summary.
Since more than 90% of the Selenium tests passed, the manual gate was skipped and the pipeline automatically promoted to the QA stage.
In the QA stage, again, 95% of Selenium tests passed. This did not meet the required 100% of test completion in the first automated gate. Since the automated gate failed, the pipeline was aborted before entering the manual gate to promote to Production.
This example shows one scenario and uses automated and manual gates to control progression of the pipeline. CloudBees Flow provides the flexibility to combine automated, procedure based, and manual gates as needed to ensure that your pipelines include the rigor required to meet your quality and compliance standards.