Example monitoring implementation with Datadog

2 minute read
On this page

CloudBees provides a reference implementation based on Datadog, which provides an example that can be used to inform you how to configure monitoring with your monitoring tool of choice.

This guide assumes Datadog agent is already installed in the Kubernetes cluster by following the Datadog documentation. CloudBees recommends using the Helm-based installation method.

The rest of this guide focus on how to set up Datadog agent to collect metrics from CloudBees CI.

Collecting metrics

operations center and managed controllers expose a set of metrics in order to monitor and understand the state of the system. To publish these metrics to Datadog, enable JMX across the cluster then configure the Datadog agent to collect the metrics available over JMX.

Agent configuration

Add the following under the path .datadog.confd in your values.yaml

jenkins.yaml: |- ad_identifiers: - jenkins init_config: is_jmx: true collect_default_metrics: true jmx_metrics: - include: bean: 'io.jenkins:type=Http,name=ActiveRequests' attribute: Count: metric_type: gauge alias: jenkins.http.activeRequests - include: bean: 'io.jenkins:type=Queue,name=Size' attribute: Value: metric_type: gauge alias: jenkins.queue.size - include: bean: 'io.jenkins:type=Queue,name=Stuck' attribute: Value: metric_type: gauge alias: jenkins.queue.stuck - include: bean: 'io.jenkins:type=Queue,name=Blocked' attribute: Value: metric_type: gauge alias: jenkins.queue.blocked - include: bean: 'io.jenkins:type=Node' attribute: Value: metric_type: gauge alias: jenkins.node.count - include: bean: 'io.jenkins:type=Job' attribute: Value: metric_type: gauge alias: jenkins.job.count - include: bean: 'io.jenkins:type=Executor' attribute: Value: metric_type: gauge alias: jenkins.executor.count # Plugins - include: bean: 'io.jenkins:type=Plugins,name=Active' attribute: Value: metric_type: gauge alias: jenkins.plugins.active - include: bean: 'io.jenkins:type=Plugins,name=Inactive' attribute: Value: metric_type: gauge alias: jenkins.plugins.inactive - include: bean: 'io.jenkins:type=Plugins,name=Failed' attribute: Value: metric_type: gauge alias: jenkins.plugins.failed - include: bean: 'io.jenkins:type=Plugins,name=WithUpdate' attribute: Value: metric_type: gauge alias: jenkins.plugins.withUpdate

then apply the configuration to the Datadog agent deployment.

This configuration file defines how metrics exported using JMX will be mapped into Datadog metrics.

YAML Manifest

Follow the Datadog documentation and provide the jenkins.yaml file described above as part of the configmap to be created instead of http-config mentioned in Datadog example.

Then in the manifest of your agent (DaemonSet/Deployment) add the following:

[...] volumeMounts: [...] - name: dd-agent-config mountPath: /conf.d volumes: [...] - name: dd-agent-config configMap: name: dd-agent-config items: - key: jenkins.yaml path: jenkins.yaml [...]

Enable JMX

In order to the Datadog agent to pick up CloudBees CI on modern cloud platforms metrics, JMX must be enabled on operations center and managed controllers requiring monitoring.

To achieve this, use Datadog autodiscovery.

operations center

Edit helm Values.yaml as follows:

OperationsCenter: Annotations: ad.datadoghq.com/jenkins.check_names: '["jenkins"]' ad.datadoghq.com/jenkins.init_configs: '[{"is_jmx": true, "collect_default_metrics": true}]' ad.datadoghq.com/jenkins.instances: '[{"jmx_url": "service:jmx:rmi:///jndi/rmi://%%host%%:1099/jmxrmi"}]' JavaOpts: >- -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false Master: JavaOpts: >- -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

This enables JMX monitoring on operations center as well as on all managed controllers.

The annotations are required in order for Datadog to autodiscover the exported metrics through JMX.

Managed controllers

The same set of annotations needs to be added to all controllers.

This can be done by browsing operations center then use Manage Jenkins  Configure System.

From there, go to the Kubernetes Controller Provisioning section  Advanced…​ button  YAML field.

Enter the following snippet:

kind: "StatefulSet" spec: template: metadata: annotations: ad.datadoghq.com/jenkins.check_names: '["jenkins"]' ad.datadoghq.com/jenkins.init_configs: '[{"is_jmx": true, "collect_default_metrics": true}]' ad.datadoghq.com/jenkins.instances: '[{"jmx_url": "service:jmx:rmi:///jndi/rmi://%%host%%:1099/jmxrmi"}]'

Then save.

You will have to edit each managed controller configuration and provide the same snippet in the yaml field.

Then restart each controller after applying the changes.

In August 2020, the Jenkins project voted to replace the term master with controller. We have taken a pragmatic approach to cleaning these up, ensuring the least amount of downstream impact as possible. CloudBees is committed to ensuring a culture and environment of inclusiveness and acceptance - this includes ensuring the changes are not just cosmetic ones, but pervasive. As this change happens, please note that the term master has been replaced through the latest versions of the CloudBees documentation with controller (as in managed controller, client controller, team controller) except when still used in the UI or in code.