DORA metrics

Analytics and metrics are essential tools in software delivery for providing visibility into the process and identifying improvement opportunities. DevOps Research and Assessment (DORA) metrics provide a way to compare development teams’ performance or DevOps maturity.

DORA metrics dashboards

CloudBees CD/RO includes two built-in dashboards; each with two widgets that calculate DORA metrics.

Access the DORA Metrics dashboards in Analytics dashboards:

DORA Metrics (Deployment Frequency, Mean Lead Time)
DORA Metrics (Mean Time To Recover, Change Failure Rate)

Each dashboard contains two widgets that calculate based on these use case descriptions:

Deployment frequency (DF) shows how often an organization successfully releases to production. It calculates the frequency of application deployments to production environments by outcome. This information is based on the results of application deployment runs. Deployment frequency measures how many application deployments run to completion over time.
Mean lead time (MLT) for changes shows the average time it takes for a commit to get into production. It calculates the average duration of pipeline runs that deliver content to production.
Mean time to recover (MTTR) shows how long it takes an organization to recover from a failure in production. It calculates the average duration of successfully completed pipeline runs tagged for incident remediation and includes an output parameter called SysID.
Change failure (CF) rate displays the percentage of deployments causing a failure in production. The change failure rate is based on the number of pipeline runs to resolve a production incident versus successful business-as-usual (non-incidents) production pipeline release pipeline runs. The output parameter SysID and an incident tag identify an incident pipeline run.

Use case examples for setting up CloudBees CD/RO to calculate DORA metrics

Use the following examples to set up CloudBees CD/RO to calculate DORA metrics for each of the four built-in widgets.

Production deployment frequency.
1. Create an Environment in CloudBees CD/RO and add a production tag value, for example, PROD.
2. Create an application in CloudBees CD/RO and map it to run on a PROD tagged environment.
3. Run the application.
4. Navigate to the DORA Metrics dashboards in Analytics dashboards.
5. Select the DORA Metrics (Deployment Frequency, Mean Lead Time) dashboard.
6. Locate the parameters input field.
  
  Figure 1. DORA Parameters
7. Enter PROD into the Production tags: parameter field to get the number of deployments grouped by date.
8. Select Apply.
Mean lead time.
1. Create or locate a release in CloudBees CD/RO.
2. Add a Tag to the release with a specific value, for example, production_release.
3. Run the production release.
4. Navigate to the DORA Metrics dashboards in Analytics dashboards.
5. Select the DORA Metrics (Deployment Frequency, Mean Lead Time) dashboard.
6. Locate the parameters input field.
7. Enter production_release to the Production tags: parameter field to get the average pipeline run duration grouped by date.
8. Select Apply.
Mean time to recovery and Change failure rate.
1. Create or locate a release in CloudBees CD/RO.
2. Add a Tag to the release with a specific value, for exmaple production_release or incident_release. This value will match a tag on the MTTR and CF dashboard.
3. Add an input parameter to the release. For this example add an input pipeline parameter called incidentID. The value of this pipeline parameter can then be used by a task procedure to set the required SysID output parameter value, where the run condition for the task is based on whether the incidentId value is set.
  
  The ServiceNow plugin procedure GetRecords_NextGen includes the SysID output parameter. If ServiceNow is not to be used, create your own procedure that sets this output parameter.
4. Run the release pipeline and enter an incident ID value.
5. Navigate to the DORA Metrics (Mean time to recover, Change failure) dashboard.
6. Locate the parameters input field.
7. Enter incident_release or production_release to the Production tags: parameter field depending on whether the release is intended for incident release or both production and incident releases.
8. Select Apply.
9. Run the release one or more times without setting the incident ID.
10. Run the release again with the incident ID set to a non-empty value.
11. Return to the MTTR and CF dashboard to view the results.

DSL example to illustrate single release

The following DSL example illustrates how to use a single release definition for both production and incident releases and will activate all the DORA metrics; once this is configured, all of the DORA dashboard widgets should contain data.

project "DORA",{
  procedure "Incident handler",{
    formalOutputParameter "SysID" // Required to identify pipeline run as an incident
    step "Set SysID output parameter",
      /*
        Command assumes "incidentId" parameter exists in the release pipeline definition.
        Command sets the output parameter SysID to the pipeline incidentId input parameter value
      */
      command: 'ectool setOutputParameter "SysID" "$[/myPipelineRuntime/incidentId]" '
  }
  release "Production and Incident",{
    tag "production_release"  // This will tag the release and any pipeline runs from it
    pipeline releaseName,{
      formalParameter "incidentId"
      stage "Production",{
        task "If Incident",
          taskType: 'PROCEDURE',
          condition: '$[incidentId]',
          subproject: projectName,
          subprocedure: "Incident handler"
        task "Deploy",
          command: 'echo deploying to production',
          subTaskType: 'DEPLOY',
          applicationProjectName:  projectName,
          applicationName:  'Sample Application',
          environmentProjectName:  projectName,
          environmentName: 'Production',
          applicationVersion: '1.0',
          artifacts: 'comp:1.1.0'
      }
    }
  }
  tagObject tag: "PROD", environmentName: 'Production'
}▼