Elasticsearch
The Elasticsearch component provides a repository for various types of CloudBees Jenkins Enterprise data, such as raw metrics, job-related information, and logs. This section provides information about the Elasticsearch component in CloudBees Jenkins Enterprise and the indices of data being persisted into it.
Data Types
CloudBees Jenkins Enterprise collects some internal log data in Elasticsearch, detailed below.
| Name Pattern | Retention Period | Contents | Disk usage expectation |
|---|---|---|---|
topbeat-* |
3 days |
Performance data for the controller/worker nodes |
70MB/node/day |
logstash-* |
7 days |
syslog entries for CloudBees Jenkins Enterprise infrastructure components |
800MB/day for the following system: three-node controller, three-node Elasticsearch, one node for master, one node for builds, 2000 jobs/day. |
Analytics data are stored in Elasticsearch using time-stamped index names. These are retained at various intervals, after which the index for that day is deleted automatically by Analytics. Future versions of Analytics will allow you to configure the retention interval of each index type. The indexes and their retention period are listed below.
| Name Pattern | Retention Period | Contents | Disk usage expectation |
|---|---|---|---|
|
3 days |
Metric reports from Jenkins masters, which are submitted every 10 seconds. |
450 mb (stable) per connected master |
|
3 years |
Hourly summary of the metrics reports for each master, based on the |
210mb per connected master, per year |
|
3 years |
Builds reported for indexing by |
2k per build |
|
3 years |
Node configuration changes reported for indexing by |
1k per configuration change |
|
Forever |
Minimal information about each job and folder. |
1k per job or folder |
|
Forever |
Dashboard and panel definitions. |
30k per dashboard |
|
Deleting a build in Jenkins does not result in the build being deleted
from Analytics, because this would diminish the accuracy and value of the
Build Analytics feature. Instead, builds will be 'forgotten' 3 years
after they were last reported — when the datestamped Note that "Reindexing" resets the lifespan of a build. So Jenkins
still has a build from 5 years ago and you Reindex that Jenkins, the
build will be retained in the |
Default Elasticsearch Configuration
| Configuration Attribute | Default Value | Description |
|---|---|---|
memory |
2048 |
Total amount of memory allocated for Elasticsearch container (2000MB). Elasticsearch, itself, gets 1024MB of heap space |
max_instance_count |
1 |
Number of Elasticsearch node |
replicas |
1 |
Number of replica shard to keep |
Recommended Configuration for Medium Size Deployment
The following recommendation is for a CloudBees Jenkins Enterprise deployment that runs about 2000 jobs a day, with three Jenkins masters and five build worker nodes. Elasticsearch is to run on three dedicated worker nodes. For AWS, it is of instance type "m4.xlarge", which has 16GB of RAM, 4 vCPUs and 200G of disk space. If OpenStack is the platform, then use the equivalent flavor.
| Configuration Attribute | Default Value | Description |
|---|---|---|
memory |
14000 |
Total amount of memory allocated for Elasticsearch container (14000MB). Elasticsearch, itself, gets 7000MB of heap space |
max_instance_count |
3 |
Number of Elasticsearch node |
replicas |
2 |
Number of shard replicas to keep |
shards |
5 |
Number of shard per index |
Changing Elasticsearch Configuration
This is typically done when the user wants to re-configure Elasticsearch, e.g, changing from the out-of-the-box configuration to one than can handle a medium size deployment described above.
Please use the following sequence of actions:
-
Add three dedicated Elasticsearch nodes by running the worker-add operation
-
Ensure that the backup has completed by querying Elasticsearch using the following URL
http://<CJE-url>/elasticsearch/_snapshot/_status. -
Execute the
elasticsearch-updateoperation. For the medium size deployment recommendation, the contents of the fileelasticsearch-update.configshould be as follows:elasticsearch-update.config[elasticsearch] max_instance_count = 3 memory = 14000 shards = 5 replicas = 2 use_dedicated_worker = true -
Restart Operations Center by executing the following commands in the CloudBees Jenkins Enterprise project directory
dna stop cjoc dna start cjoc
CloudBees Jenkins Enterprise Elasticsearch Operations
The following sections are specific to those operations that affect Elasticsearch running in the CloudBees Jenkins Enterprise environment.
Backing Up Elasticsearch
elasticsearch-backupThis operation initiates a back-up action for all Elasticsearch indices into a repository called "tiger-backup". This operation is invoked asynchronously, i.e., it returns immediately before the back-up action has finished.
To check the back-up action status, reference the following Elasticsearch
API end-point http://<CJE-url>/elasticsearch/_snapshot/tiger-backup/_status.
|
Restoring Elasticsearch
elasticsearch-restoreThis operation is the reverse of the elasticsearch-backup operation. It is also an asynchronous
operation and the same API end-point can be used to check for its completion status.
Deleting Elasticsearch Indices
elasticsearch-indices-deleteThis operations removes older logstash-* and topbeat-* indices. By default, it will remove data older than three days. However, the user can customize the cutoff point, by modifying the file "elasticsearch-indices-delete.config".
[elasticsearch]
## The number of days worth of elasticsearch indices to keep.
#days_to_keep = 3