Example monitoring implementation with Datadog

CloudBees provides a reference implementation based on Datadog, which provides an example that can be used to inform you how to configure monitoring with your monitoring tool of choice.

This guide assumes Datadog agent is already installed in the Kubernetes cluster by following the Datadog documentation. CloudBees recommends using the Helm-based installation method.

The rest of this guide focus on how to set up Datadog agent to collect metrics from CloudBees CI.

Collecting metrics

Operations centers and managed controllers expose a set of metrics in order to monitor and understand the state of the system. To publish these metrics to Datadog, enable JMX across the cluster then configure the Datadog agent to collect the metrics available over JMX.

Agent configuration

Helm (recommended)

Add the following under the path .datadog.confd in your values.yaml

jenkins.yaml: |-
      ad_identifiers:
          - jenkins
      init_config:
          is_jmx: true
          collect_default_metrics: true
      jmx_metrics:
          - include:
              bean: 'io.jenkins:type=Http,name=ActiveRequests'
              attribute:
                Count:
                  metric_type: gauge
                  alias: jenkins.http.activeRequests
          - include:
              bean: 'io.jenkins:type=Queue,name=Size'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.queue.size
          - include:
              bean: 'io.jenkins:type=Queue,name=Stuck'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.queue.stuck
          - include:
              bean: 'io.jenkins:type=Queue,name=Blocked'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.queue.blocked
          - include:
              bean: 'io.jenkins:type=Node'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.node.count
          - include:
              bean: 'io.jenkins:type=Job'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.job.count
          - include:
              bean: 'io.jenkins:type=Executor'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.executor.count
          # Plugins
          - include:
              bean: 'io.jenkins:type=Plugins,name=Active'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.plugins.active
          - include:
              bean: 'io.jenkins:type=Plugins,name=Inactive'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.plugins.inactive
          - include:
              bean: 'io.jenkins:type=Plugins,name=Failed'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.plugins.failed
          - include:
              bean: 'io.jenkins:type=Plugins,name=WithUpdate'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.plugins.withUpdate▼

then apply the configuration to the Datadog agent deployment.

This configuration file defines how metrics exported using JMX will be mapped into Datadog metrics.

YAML Manifest

Follow the Datadog documentation and provide the jenkins.yaml file described above as part of the configmap to be created instead of http-config mentioned in Datadog example.

Then in the manifest of your agent (DaemonSet/Deployment) add the following:

[...]
        volumeMounts:
        [...]
          - name: dd-agent-config
            mountPath: /conf.d
      volumes:
      [...]
        - name: dd-agent-config
          configMap:
            name: dd-agent-config
            items:
            - key: jenkins.yaml
              path: jenkins.yaml
[...]▼

Enable JMX

In order to the Datadog agent to pick up CloudBees CI on modern cloud platforms metrics, JMX must be enabled on operations center and managed controllers requiring monitoring.

To achieve this, use Datadog autodiscovery.

Operations center

Edit helm Values.yaml as follows:

OperationsCenter:
  Annotations:
    ad.datadoghq.com/jenkins.check_names: '["jenkins"]'
    ad.datadoghq.com/jenkins.init_configs: '[{"is_jmx": true, "collect_default_metrics": true}]'
    ad.datadoghq.com/jenkins.instances: '[{"jmx_url": "service:jmx:rmi:///jndi/rmi://%%host%%:1099/jmxrmi"}]'
  JavaOpts: >-
    -Dcom.sun.management.jmxremote.port=1099
    -Dcom.sun.management.jmxremote.authenticate=false
    -Dcom.sun.management.jmxremote.ssl=false
Master:
  JavaOpts: >-
    -Dcom.sun.management.jmxremote.port=1099
    -Dcom.sun.management.jmxremote.authenticate=false
    -Dcom.sun.management.jmxremote.ssl=false▼

This enables JMX monitoring on operations center as well as on all managed controllers.

The annotations are required in order for Datadog to autodiscover the exported metrics through JMX.

Managed controllers

The same set of annotations needs to be added to all controllers.

This can be done by browsing operations center then use Manage Jenkins Configure System.

From there, go to the Kubernetes Controller Provisioning section Advanced… button YAML field.

Enter the following snippet:

kind: "StatefulSet"
spec:
  template:
    metadata:
      annotations:
        ad.datadoghq.com/jenkins.check_names: '["jenkins"]'
        ad.datadoghq.com/jenkins.init_configs: '[{"is_jmx": true, "collect_default_metrics": true}]'
        ad.datadoghq.com/jenkins.instances: '[{"jmx_url": "service:jmx:rmi:///jndi/rmi://%%host%%:1099/jmxrmi"}]'▼

Then save.

You will have to edit each managed controller configuration and provide the same snippet in the yaml field.

Then restart each controller after applying the changes.

In August 2020, the Jenkins project voted to replace the term master with controller. We have taken a pragmatic approach to cleaning these up, ensuring the least amount of downstream impact as possible. CloudBees is committed to ensuring a culture and environment of inclusiveness and acceptance - this includes ensuring the changes are not just cosmetic ones, but pervasive. As this change happens, please note that the term master has been replaced through the latest versions of the CloudBees documentation with controller (as in managed controller, client controller, team controller) except when still used in the UI or in code.