Configuring the CloudBees Analytics server

DORA metrics

3 minute readDeveloper productivityData analytics

Analytics and metrics are important tools in software delivery for providing visibility into the process and identifying opportunities for improvements. DevOps Research and Assessment (DORA) metrics provide a way to compare the performance or DevOps maturity of development teams. This information can be useful for tracking improvements in team performance.

DORA metrics are calculated based on deployment and pipeline (standalone or release) runs. Tags are used to distinguish relevant runs for particular DORA metrics. Tags are attached to the runs, and the dashboards use parameters to make the association.

You can apply tags to the application (for deployments) and pipeline (or release) definitions. Tags associated with application or pipelines are automatically attached to their runtime objects. Alternatively, you can apply tags at runtime. For example, for Mean Time to Recovery, you can create an optional parameter, such as incidentId, for an incident ticket on a production release pipeline. If this value is present due to an incident, you can apply the incident tag to the pipeline runtime using a task with a run condition set to $[incidentId]. The command is ectool tagObject --flowRuntimeId "$[/myPipelineRuntime/flowRuntimeId]" --tags incidentRelease.

The following DORA metrics are available in dashboards:

  • Deployment frequency shows how often an organization successfully releases to production. It calculates the frequency of application deployments to production environments by outcome. This information is based on the results of application deployment runs. Deployment frequency measures how many application deployments run to completion over time.

  • Mean lead time for changes shows the average amount of time it takes for a commit to get into production. It calculates the average duration of pipeline runs that deliver content to production.

  • Mean time to recover shows how long it takes an organization to recover from a failure in production. It calculates the average duration of successfully completed pipeline runs that have been tagged for incident remediation.

  • Change failure rate displays the percentage of deployments causing a failure in production. Change failure rate is based on the number of pipeline runs to resolve a production issue versus successful standard business-as-usual non-issue production pipeline runs. A production issue pipeline run is identified by whether it has been tagged as remediating an incident.

Use cases for setting up CloudBees CD/RO to calculate DORA metrics

Use the following information as a guideline for setting up your environment to enable CloudBees CD/RO to calculate DORA metrics.

  • Production deployment frequency

    1. Configure an Environment in CloudBees CD/RO and tag it with specific tag value, for example, PROD.

    2. Configure an Application in CloudBees CD/RO and map it to run on a PROD environment.

    3. Run the application.

    4. Navigate to the DORA Metrics (Deployment Frequency, Mean Lead Time) dashboard, and then submit PROD as the Production tags: parameter value to get the application deployments on the PROD environment grouped by dates and deployment outcome.

  • Mean lead time

    1. Configure a pipeline in CloudBees CD/RO and tag it with specific tag value, for example production_pipeline.

    2. Run the production pipeline.

    3. Navigate to the DORA Metrics (Deployment Frequency, Mean Lead Time) dashboard, and then submit production_pipeline as the Production tags: parameter value to get the average pipeline run duration grouped by date.

  • Mean time to recovery and change failure rate

    1. Create a release and tag it, for example, incidentRelease, or use the mechanism described above that uses an incident ticket parameter to tag the pipeline run.

    2. Add a GetRecords_NextGen task to the release pipeline.

      The task type must be plugin, and you must use EC-ServiceNow. The only supported procedure is GetRecords_NextGen.
    3. Set the task parameter values. Valid values are:

      • Table: incident

        This value is case-sensitive.
      • Retrieval Type: The system ID number

      • Numbers: The valid ServiceNow incident number or system ID

    4. Run the release pipeline.

    5. The pipeline collects incident details and pushes reporting data.

    6. The pipeline waits for the incident to be closed and the pipeline completes. Recovery time is the pipeline duration in this case.

    7. From the CloudBees CD/RO main menu, select Analytics  Dashboards.

    8. From the Dashboards list, select DORA Metrics (Mean Time to Recover, Change Failure Rate).

    9. Select Parameters, then enter your tag (for example, incidentRelease) in Incident tags.

    10. Select Apply.