System health monitoring

2 minute readReference

CloudBees CD/RO is instrumented to publish system health metrics to a StatsD server via UDP. Once enabled in the CloudBees CD/RO server, use a visualization tool such as Grafana to view system health data.

You are responsible for standing up and maintaining your own StatsD server, Graphite database, and Grafana UI. Go to the following sites for documentation and tutorials:

  • https://github.com/etsy/statsd/

  • https://graphite.readthedocs.io/en/stable/

  • https://www.digitalocean.com/community/tutorials/an-introduction-to-tracking-statistics-with-graphite-statsd-and-collectd/

  • https://www.digitalocean.com/community/tutorials/how-to-configure-statsd-to-collect-arbitrary-stats-for-graphite-on-ubuntu-14-04

Try out system health monitoring

To get started, configure system health monitoring on your CloudBees CD/RO server by following the steps in configuring system health monitoring. Then, stand up a StatsD/Graphite/Grafana stack using this Docker image:

  1. Clone the image.

  2. Run make up.

  3. Import the Grafana dashboard found here.

  4. Monitor CloudBees CD/RO server events using different panels provided in the dashboard.

Configuring system health monitoring

You must enable system health monitoring on the CloudBees CD/RO server in order for it to send metrics to the StatsD server. It may take up to two hours before data can be visualized.

Via the UI
  1. Navigate to the System health monitoring configuration page: from the main menu select Administration ConfigurationsSystem health monitoring.

  2. Supply the following data:

    NameDescription

    Enabled

    If checked, StatsD metric generation is enabled. Default is disabled.

    Host Name

    The host name or IP address of the StatsD server.

    Include Host Name

    If checked, send the CloudBees CD/RO host name generating the metrics. Default is enabled.

    Port

    The port on which StatsD metrics are sent. Default is 8125.

    Prefix

    The prefix added to every StatsD metric name. Use to differentiate from other metrics sent to the same StatsD server. Default is CloudBeesCD.

    Flush rate

    The rate at which the StatsD client flushes pending metric values to the listener. For example, if set to 0.1, 1 in 10 writes flush the pending values. Sample rate

Via the command line

Using ectool on the CloudBees CD/RO server system command line, issue the following command:

setStatsdConfiguration
    [--enabled <0|1|true|false>]
    [--hostName <hostName>]
    [--includeHostName <0|1|true|false>]
    [--port <port>]
    [--prefix <prefix>]
    [--flushRate <flushRate>
    [--sampleRate <sampleRate>]

For example,

ectool --server localhost setStatsdConfiguration \
       --enabled 1 \
       --hostName sv-stasd-grafana \
       --includeHostName 1 \
       --port 8126 \
       --prefix CloudBeesCD \
       --flushRate 1.0 \
       --sampleRate 0.75

See setStatsConfiguration for further details.

StatsD buckets

Out of the different StatsD metric types as listed here, CloudBees CD/RO server aggregates metrics into the following buckets:

Counter metrics

CloudBeesCD.*.hibernate
CloudBeesCD.*.jobSteps
CloudBeesCD.*.jobs
    CloudBeesCD.jobs.runnableSteps
CloudBeesCD.*.login.*
CloudBeesCD.*.loginSaml
CloudBeesCD.*.logout
CloudBeesCD.*.memory
    CloudBeesCD.*.memory.gc.G1_Old_Generation.collectionCount
CloudBeesCD.*.messageService.*
CloudBeesCD.*.stepScheduler

Gauge metrics

CloudBeesCD.*.api.active
CloudBeesCD.*.memory.*
    CloudBeesCD.*.memory.G1_Old_Gen.usage.committed
    CloudBeesCD.*.memory.G1_Old_Gen.usage.max
CloudBeesCD.*.cpu
    CloudBeesCD.*.cpu.user
    CloudBeesCD.*.cpu.system
CloudBeesCD.*.files
CloudBeesCD.*.stepScheduler.category
CloudBeesCD.*.licenses

Timer metrics

CloudBeesCD.*.api
CloudBeesCD.*.hibernate
    CloudBeesCD.*.hibernate.apiRetryDelays
    CloudBeesCD.*.hibernate.transactionRetryDelay
CloudBeesCD.*.memory
    CloudBeesCD.*.memory.gc.G1_Old_Generation.collectionTime
CloudBeesCD.*.messageService
    CloudBeesCD.*.messageService.deferredMessageDelay
    CloudBeesCD.*.messageService.deferredMessages

These are the basic names with the prefix CloudbeesCD, defined at configuration time, appended. Further append them with any naming hierarchy required by the particular StatsD server provider.

Using the metrics

Configure a dashboard to visualize data collected by the StatsD server. Get a sample Grafana dashboard here.