Elasticsearch basics

4 minute readData analytics

The Elasticsearch database stores all data for use by CloudBees Analytics reports and dashboards. While developing reports and dashboards, it is useful to view the data available in the Elasticsearch database. Information in this section helps you to set up and examine data and to run reports and to export to other systems.

Examining data in the Elasticsearch database

This section helps you to set up and directly query the Elasticsearch database. It can be useful to examine report data and to run reports directly against the Elasticsearch database for debugging purposes when you create your own reports and dashboards.

Run a report

This section demonstrates how to manually submit a CloudBees Analytics report to CloudBees Analytics engine and to the Elasticsearch database. It assumes you have deployment data in your database.

Define the report

Use the CloudBees Analytics UI to define search criteria.

  1. Select Analytics  Reports. The reports list displays.

  2. Select the Build outcome report. This is the custom report you previously created. The Report editor displays the report in advanced mode and assumes you have deployment data in your database.

  3. Select Tabular preview to run the report and view the result set.

You can toggle between the Report editor to edit the report and Tabular preview to view the results. Select Save when you are satisfied with the report criteria.

To change a report bundled with CloudBees Analytics, you must first copy it and change the copy.

Issuing CloudBees Analytics API requests

Use the runReports CloudBees Analytics API command to run a report. The example below uses ectool to issue the command that runs the Build outcome report.

ectool runReport "CloudBees" "Build outcome"

Results similar to the following are returned:

<response requestId="1" nodeId="192.168.5.138"> <result> <deployment_outcome_count>26</deployment_outcome_count> <total>26</total> <deployment_outcome>success</deployment_outcome> <avg_deployment_duration>530769.23</avg_deployment_duration> </result> </response>

CloudBees Analytics API commands are also available from the CloudBees CD/RO Swagger UI available via https://<cloudbees-cd-server_hostname>/rest/doc/v1.0/.

Issuing Elasticsearch API requests

This section shows you how to issue Elasticsearch API requests to the database via cURL.

The Elasticsearch version shipped with CloudBees Analytics version 10.2 has been updated from v6.6.2 to v7.10.2. This update may create breaking changes in your custom reports. All changes related to the new version are described in the Elasticsearch documentation. The following change may affect your reports the most.

Accessing missing document values returns an error. The doc['field'].value returns an exception if the document is missing a value for the field field. To check if a document is missing a value, you can use doc['field'].size() == 0.

Prerequisites

  • Elasticsearch database login credentials. Defaults are:

    • user: reportinguser

    • password: changeme

  • A tool for issuing REST POST requests. For example, cURL

  • A tool for Base64 encoding

Accessing the Elasticsearch database

From a web browser, access the Elasticsearch database with the URL https://<hostname>:9200, where <hostname> is the same hostname as your CloudBees Analytics server. Alternatively, navigate to the Administration  Configurations  Analytics server to determine your Elasticsearch database URL.

Enter your credentials for Elasticsearch in the sign-in dialog; confirm you get a response similar to this:

{ "name" : "<hostname>", "cluster_name" : "elasticsearch", "cluster_uuid" : "nbV6hwg0QM-ICIIGcvK2AQ", "version" : { "number" : "6.6.2", "build_flavor" : "oss", "build_type" : "tar", "build_hash" : "3bd3e59", "build_date" : "2019-03-06T15:16:26.864148Z", "build_snapshot" : false, "lucene_version" : "7.6.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }

Base64 encoded credentials

Your credentials are required to be in a Base64 encoded format. To generate this encoded credential string you can use ectool or an online tool, the latter is not recommended for production use since you cannot be assured you data is held privately by an unknown party.

ectool evalDsl '"reportinguser:changeme".bytes.encodeBase64().toString()'
<response requestId="1" nodeId="10.1.1.101"> <value>cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ==</value> </response>
The resulting key is `cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ`.

Create the request

The following cURL request queries the ef-build table in the database. In the context of the URL below, ef-build-* tells Elasticsearch to return data from the CloudBees Analytics (ef-) deployment table. The asterisk is a wildcard indicating to return deployment data for all time; the indices are tagged with the year (-2021).

curl -X POST \ 'https://<hostname>:9200/ef-build-*/_search?pretty=' \ -H 'Authorization: Basic cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ ' \ -H 'Content-Type: application/json' \ -d '{"size": 10,"query": {"bool": {}}}'

Assuming the CloudBees Analytics server instance has deployment data, running this request should return something like the following (only the first few lines are shown here). Report data is returned in _source elements in the result set. Refer to schemas documented in Understanding the CloudBees Analytics data model for description of result set fields. In this case, Understanding the CloudBees Analytics data model.

{ { "took" : 8, "timed_out" : false, "_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1624, "max_score" : 1.0, "hits" : [ { "_index" : "ef-build-2021", "_type" : "doc", "_id" : "5CVyPXcBLrhx_rZ26wl7", "_score" : 1.0, "_source" : { "timestamp" : "2021-01-26T06:47:55.751Z", "reportObjectType" : "build", "controllerName" : "inst2-test_9a715976-cf41-43c8-9429-94b0099523ce", "startTime" : "2021-01-26T06:46:55.751Z", "duration" : 60000, "buildStatus" : "SUCCESS", ...

Search the database

The following cURL request queries the ef-build-* table in the database searching for records whose buildStatus field is set to SUCCESS. In the context of the URL below, ef-build-* tells Elasticsearch to return data from the CloudBees Analytics (ef-) deployment table. The asterisk is a wildcard indicating to return deployment data for all time; the indices are tagged with the year (-2021).

curl -X POST \ 'https://<hostname>:9200/ef-deployment%2A/_search?q=deploymentOutcome:success' \ -H 'Authorization: Basic cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ ' \ -H 'Content-Type: application/json'

Assuming the instance has been used for deployments, running this request should return something like the following (only the first few lines are shown here). The field in the request is highlighted in blue.

{ { "took" : 8, "timed_out" : false, "_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1624, "max_score" : 1.0, "hits" : [ { "_index" : "ef-build-2021", "_type" : "doc", "_id" : "5CVyPXcBLrhx_rZ26wl7", "_score" : 1.0, "_source" : { "timestamp" : "2021-01-26T06:47:55.751Z", "reportObjectType" : "build", "controllerName" : "inst2-test_9a715976-cf41-43c8-9429-94b0099523ce", "startTime" : "2021-01-26T06:46:55.751Z", "duration" : 60000, "buildStatus" : "SUCCESS", ...

Elasticsearch keywords

Since report definitions are constructed using Elasticsearch DSL queries behind the scenes, some knowledge of Elasticsearch is valuable to build custom reports. Extensive documentation and examples are available on elastic.co. Elasticsearch training is beyond the scope of this document.