System health monitoring

2 minute readReference

CloudBees CD/RO is instrumented to publish system health metrics to a StatsD server via UDP. Once enabled in the CloudBees CD/RO server, use a visualization tool such as Grafana to view system health data.

You are responsible for standing up and maintaining your own StatsD server, Graphite database, and Grafana UI. Go to the following sites for documentation and tutorials:

  • https://github.com/etsy/statsd/

  • https://graphite.readthedocs.io/en/stable/

  • https://www.digitalocean.com/community/tutorials/an-introduction-to-tracking-statistics-with-graphite-statsd-and-collectd/

  • https://www.digitalocean.com/community/tutorials/how-to-configure-statsd-to-collect-arbitrary-stats-for-graphite-on-ubuntu-14-04

Try out system health monitoring

To get started, configure system health monitoring on your CloudBees CD/RO server by following the steps in configuring system health monitoring. Then, stand up a StatsD/Graphite/Grafana stack using this Docker image:

  1. Clone the image.

  2. Run make up.

  3. Import the Grafana dashboard found here.

  4. Monitor CloudBees CD/RO server events using different panels provided in the dashboard.

Configuring system health monitoring

You must enable system health monitoring on the CloudBees CD/RO server for it to send metrics to the StatsD server. It may take up to two hours before data can be visualized.

Via the UI
  1. Navigate to the System health monitoring configuration page: from the main menu, select Administration  Configurations  System health monitoring.

  2. Enter the following data:

    Name Description

    Enabled

    If checked, StatsD metric generation is enabled. Default is disabled.

    Host Name

    The host name or IP address of the StatsD server.

    Include Host Name

    If checked, send the CloudBees CD/RO host name generating the metrics. Default is enabled.

    Port

    The port on which StatsD metrics are sent. Default is 8125.

    Prefix

    The prefix added to every StatsD metric name. Use to differentiate from other metrics sent to the same StatsD server. Default is CloudBeesCD.

    Flush rate

    The rate at which the StatsD client flushes pending metric values to the listener. For example, if set to 0.1, 1 in 10 writes flush the pending values. Sample rate

Via the command line

Using ectool on the CloudBees CD/RO server system command line, issue the following command:

setStatsdConfiguration [--enabled <0|1|true|false>] [--hostName <hostName>] [--includeHostName <0|1|true|false>] [--port <port>] [--prefix <prefix>] [--flushRate <flushRate> [--sampleRate <sampleRate>]

For example,

ectool --server localhost setStatsdConfiguration \ --enabled 1 \ --hostName sv-stasd-grafana \ --includeHostName 1 \ --port 8126 \ --prefix CloudBeesCD \ --flushRate 1.0 \ --sampleRate 0.75

Refer to setStatsConfiguration for further details.

StatsD buckets

Out of the different StatsD metric types as listed here, CloudBees CD/RO server aggregates metrics into the following buckets:

Counter metrics

CloudBeesCD.*.hibernate CloudBeesCD.*.jobSteps CloudBeesCD.*.jobs CloudBeesCD.jobs.runnableSteps CloudBeesCD.*.login.* CloudBeesCD.*.loginSaml CloudBeesCD.*.logout CloudBeesCD.*.memory CloudBeesCD.*.memory.gc.G1_Old_Generation.collectionCount CloudBeesCD.*.messageService.* CloudBeesCD.*.stepScheduler

Gauge metrics

CloudBeesCD.*.api.active CloudBeesCD.*.memory.* CloudBeesCD.*.memory.G1_Old_Gen.usage.committed CloudBeesCD.*.memory.G1_Old_Gen.usage.max CloudBeesCD.*.cpu CloudBeesCD.*.cpu.user CloudBeesCD.*.cpu.system CloudBeesCD.*.files CloudBeesCD.*.stepScheduler.category CloudBeesCD.*.licenses

Timer metrics

CloudBeesCD.*.api CloudBeesCD.*.hibernate CloudBeesCD.*.hibernate.apiRetryDelays CloudBeesCD.*.hibernate.transactionRetryDelay CloudBeesCD.*.memory CloudBeesCD.*.memory.gc.G1_Old_Generation.collectionTime CloudBeesCD.*.messageService CloudBeesCD.*.messageService.deferredMessageDelay CloudBeesCD.*.messageService.deferredMessages

These are the basic names with the prefix CloudbeesCD, defined at configuration time, appended. Further append them with any naming hierarchy required by the particular StatsD server provider.

Using the metrics

Configure a dashboard to visualize data collected by the StatsD server. Get a sample Grafana dashboard here.