Example monitoring implementation with Datadog

On this page

This guide assumes Datadog agent is already installed in the Kubernetes cluster by following the Datadog documentation. CloudBees recommends using the helm-based installation method.

The rest of this guide focus on how to set up Datadog agent to collect metrics from CloudBees Core.

Collecting metrics

Operations Center and Managed Masters expose a set of metrics in order to monitor and understand the state of the system. In order to publish these metrics to Datadog, we’ll enable JMX across the cluster then configure the Datadog agent to collect the metrics available over JMX.

Agent configuration

Add the following under the path .datadog.confd in your values.yaml

jenkins.yaml: |-
      ad_identifiers:
          - jenkins
      init_config:
          is_jmx: true
          collect_default_metrics: true
      jmx_metrics:
          - include:
              bean: 'io.jenkins:type=Http,name=ActiveRequests'
              attribute:
                Count:
                  metric_type: gauge
                  alias: jenkins.http.activeRequests
          - include:
              bean: 'io.jenkins:type=Queue,name=Size'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.queue.size
          - include:
              bean: 'io.jenkins:type=Queue,name=Stuck'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.queue.stuck
          - include:
              bean: 'io.jenkins:type=Queue,name=Blocked'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.queue.blocked
          - include:
              bean: 'io.jenkins:type=Node'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.node.count
          - include:
              bean: 'io.jenkins:type=Job'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.job.count
          - include:
              bean: 'io.jenkins:type=Executor'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.executor.count
          # Plugins
          - include:
              bean: 'io.jenkins:type=Plugins,name=Active'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.plugins.active
          - include:
              bean: 'io.jenkins:type=Plugins,name=Inactive'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.plugins.inactive
          - include:
              bean: 'io.jenkins:type=Plugins,name=Failed'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.plugins.failed
          - include:
              bean: 'io.jenkins:type=Plugins,name=WithUpdate'
              attribute:
                Value:
                  metric_type: gauge
                  alias: jenkins.plugins.withUpdate

then apply the configuration to the Datadog agent deployment.

This configuration file defines how metrics exported using JMX will be mapped into Datadog metrics.

YAML Manifest

Follow the Datadog documentation and provide the jenkins.yaml file described above as part of the configmap to be created instead of http-config mentioned in Datadog example.

Then in the manifest of your agent (DaemonSet/Deployment) add the following:

[...]
        volumeMounts:
        [...]
          - name: dd-agent-config
            mountPath: /conf.d
      volumes:
      [...]
        - name: dd-agent-config
          configMap:
            name: dd-agent-config
            items:
            - key: jenkins.yaml
              path: jenkins.yaml
[...]

Enable JMX

In order to the Datadog agent to pick up CloudBees Core on modern cloud platforms metrics, JMX must be enabled on Operations Center and Managed Masters requiring monitoring.

To achieve this, we will use Datadog autodiscovery.

Operations Center

Edit cloudbees-core.yaml as follow:

diff --git a/cjoc.yml b/cjoc.yml
index 152fa6d..d631f02 100644
--- a/cjoc.yml
+++ b/cjoc.yml
@@ -31,6 +31,10 @@ spec:
       labels:
         com.cloudbees.cje.type: cjoc
         com.cloudbees.cje.tenant: cjoc
+      annotations:
+        ad.datadoghq.com/jenkins.check_names: '["jenkins"]'
+        ad.datadoghq.com/jenkins.init_configs: '[{"is_jmx": true, "collect_default_metrics": true}]'
+        ad.datadoghq.com/jenkins.instances: '[{"jmx_url": "service:jmx:rmi:///jndi/rmi://%%host%%:1099/jmxrmi"}]'
     spec:
       serviceAccountName: cjoc
       terminationGracePeriodSeconds: 10
@@ -42,6 +46,12 @@ spec:
           value: KUBERNETES
         - name: JENKINS_OPTS
           value: --prefix=/cjoc
+        - name: MASTER_GLOBAL_JAVA_OPTIONS
+          value: >-
+            -Dcom.sun.management.jmxremote=true
+            -Dcom.sun.management.jmxremote.port=1099
+            -Dcom.sun.management.jmxremote.ssl=false
+            -Dcom.sun.management.jmxremote.authenticate=false
         - name: JAVA_OPTS
           # To allocate masters using a non-default storage class, add the following
           # -Dcom.cloudbees.masterprovisioning.kubernetes.KubernetesMasterProvisioning.storageClassName=some-storage-class
@@ -60,6 +70,10 @@ spec:
             -Dcom.cloudbees.jce.masterprovisioning.DockerImageDefinitionConfiguration.disableAutoConfiguration=true
             -Dcom.cloudbees.jce.masterprovisioning.DockerImageDefinitionConfiguration.masterImageName="CloudBees Core - Managed Master VERSION"
             -Dcom.cloudbees.jce.masterprovisioning.DockerImageDefinitionConfiguration.masterImage=cloudbees/cloudbees-core-mm:version
+            -Dcom.sun.management.jmxremote=true
+            -Dcom.sun.management.jmxremote.port=1099
+            -Dcom.sun.management.jmxremote.ssl=false
+            -Dcom.sun.management.jmxremote.authenticate=false
         ports:
         - containerPort: 8080
         - containerPort: 50000

This enables JMX monitoring on Operations Center as well as on all Managed Masters.

The annotations are required in order for Datadog to autodiscover the exported metrics through JMX.

Managed Masters

The same set of annotations needs to be added to all masters.

This can be done by browsing Operations Center then use menu:Manage Jenkins[Configure System].

From there, go to the menu:Kubernetes Master Provisioning section[Advanced…​ button > YAML field].

Enter the following snippet:

kind: "StatefulSet"
spec:
  template:
    metadata:
      annotations:
        ad.datadoghq.com/jenkins.check_names: '["jenkins"]'
        ad.datadoghq.com/jenkins.init_configs: '[{"is_jmx": true, "collect_default_metrics": true}]'
        ad.datadoghq.com/jenkins.instances: '[{"jmx_url": "service:jmx:rmi:///jndi/rmi://%%host%%:1099/jmxrmi"}]'

Then save.

You will have to edit each managed master configuration and provide the same snippet in the menu:yaml field[].

Then restart each master after applying the changes.