Elasticsearch

The Elasticsearch component provides a repository for various types of CloudBees Jenkins Enterprise data, such as raw metrics, job-related information, and logs. This section provides information about the Elasticsearch component in CloudBees Jenkins Enterprise and the indices of data being persisted into it.

Data Types

CloudBees Jenkins Enterprise collects some internal log data in Elasticsearch, detailed below.

Table 1. CloudBees Jenkins Enterprise-internal data collected
Name Pattern	Retention Period	Contents	Disk usage expectation
topbeat-*	3 days	Performance data for the controller/worker nodes	70MB/node/day
logstash-*	7 days	syslog entries for CloudBees Jenkins Enterprise infrastructure components	800MB/day for the following system: three-node controller, three-node Elasticsearch, one node for master, one node for builds, 2000 jobs/day.

Analytics data are stored in Elasticsearch using time-stamped index names. These are retained at various intervals, after which the index for that day is deleted automatically by Analytics. Future versions of Analytics will allow you to configure the retention interval of each index type. The indexes and their retention period are listed below.

Table 2. Index Purpose and Retention Period
Name Pattern	Retention Period	Contents	Disk usage expectation
`metrics-*`	3 days	Metric reports from Jenkins masters, which are submitted every 10 seconds.	450 mb (stable) per connected master
`metrics-hourly-*`	3 years	Hourly summary of the metrics reports for each master, based on the `metrics-*` index.	210mb per connected master, per year
`builds-*`	3 years	Builds reported for indexing by `Client Masters`. Contains minimal data fields needed for reporting. Does not include console logs or artifacts. Starting from CloudBees Jenkins Analytics 1.8.100 does not include detailed info about nodes, this data has been moved to the `nodes-*` index.	2k per build
`nodes-*`	3 years	Node configuration changes reported for indexing by `Client Masters`. Event examples: node config save, creation/deletion of nodes. Such data will be recorded for all node types including ephemeral and cloud-provisioned nodes.	1k per configuration change
`items`	Forever	Minimal information about each job and folder.	1k per job or folder
`kibana-4-cloudbees`	Forever	Dashboard and panel definitions.	30k per dashboard

Deleting a build in Jenkins does not result in the build being deleted from Analytics, because this would diminish the accuracy and value of the Build Analytics feature. Instead, builds will be 'forgotten' 3 years after they were last reported — when the datestamped build-* for that day is deleted.

Note that "Reindexing" resets the lifespan of a build. So Jenkins still has a build from 5 years ago and you Reindex that Jenkins, the build will be retained in the builds-* index for another three years, regardless of its lifespan in Jenkins itself.

Default Elasticsearch Configuration

Configuration Attribute	Default Value	Description
memory	2048	Total amount of memory allocated for Elasticsearch container (2000MB). Elasticsearch, itself, gets 1024MB of heap space
max_instance_count	1	Number of Elasticsearch node
replicas	1	Number of replica shard to keep

Configuration Attribute

Default Value

Description

memory

2048

Total amount of memory allocated for Elasticsearch container (2000MB). Elasticsearch, itself, gets 1024MB of heap space

max_instance_count

Number of Elasticsearch node

replicas

Number of replica shard to keep

Recommended Configuration for Medium Size Deployment

The following recommendation is for a CloudBees Jenkins Enterprise deployment that runs about 2000 jobs a day, with three Jenkins masters and five build worker nodes. Elasticsearch is to run on three dedicated worker nodes. For AWS, it is of instance type "m4.xlarge", which has 16GB of RAM, 4 vCPUs and 200G of disk space. If OpenStack is the platform, then use the equivalent flavor.

Configuration Attribute	Default Value	Description
memory	14000	Total amount of memory allocated for Elasticsearch container (14000MB). Elasticsearch, itself, gets 7000MB of heap space
max_instance_count	3	Number of Elasticsearch node
replicas	2	Number of shard replicas to keep
shards	5	Number of shard per index

Configuration Attribute

Default Value

Description

memory

14000

Total amount of memory allocated for Elasticsearch container (14000MB). Elasticsearch, itself, gets 7000MB of heap space

max_instance_count

Number of Elasticsearch node

replicas

Number of shard replicas to keep

shards

Number of shard per index

Changing Elasticsearch Configuration

This is typically done when the user wants to re-configure Elasticsearch, e.g, changing from the out-of-the-box configuration to one than can handle a medium size deployment described above.

Please use the following sequence of actions:

Add three dedicated Elasticsearch nodes by running the worker-add operation
Backup Elasticsearch
Ensure that the backup has completed by querying Elasticsearch using the following URL http://<CJE-url>/elasticsearch/_snapshot/_status.
Execute the elasticsearch-update operation. For the medium size deployment recommendation, the contents of the file elasticsearch-update.config should be as follows:
elasticsearch-update.config
```
[elasticsearch]
max_instance_count = 3
memory = 14000
shards = 5
replicas = 2
use_dedicated_worker = true
```
Restore Elasticsearch indices from the backup in step 2
Restart operations center by executing the following commands in the CloudBees Jenkins Enterprise project directory
```
dna stop cjoc
dna start cjoc
```

CloudBees Jenkins Enterprise Elasticsearch Operations

The following sections are specific to those operations that affect Elasticsearch running in the CloudBees Jenkins Enterprise environment.

Backing Up Elasticsearch

elasticsearch-backup

This operation initiates a back-up action for all Elasticsearch indices into a repository called "tiger-backup". This operation is invoked asynchronously, i.e., it returns immediately before the back-up action has finished.

To check the back-up action status, reference the following Elasticsearch API end-point http://<CJE-url>/elasticsearch/_snapshot/tiger-backup/_status.

Restoring Elasticsearch

elasticsearch-restore

This operation is the reverse of the elasticsearch-backup operation. It is also an asynchronous operation and the same API end-point can be used to check for its completion status.

Deleting Elasticsearch Indices

elasticsearch-indices-delete

This operations removes older logstash-* and topbeat-* indices. By default, it will remove data older than three days. However, the user can customize the cutoff point, by modifying the file "elasticsearch-indices-delete.config".

[elasticsearch]

## The number of days worth of elasticsearch indices to keep.
#days_to_keep = 3