Elasticsearch basics

Data analytics

The Elasticsearch database stores all data for use by CloudBees Analytics reports and dashboards. While developing reports and dashboards, it is useful to see what data is available in the Elasticsearch database. Information in this section helps you to set up and examine data and to run reports and to export to other systems.

Examining data in the Elasticsearch database

This section helps you to set up and directly query the Elasticsearch database. It can be useful to examine report data and to run reports directly against the Elasticsearch database for debugging purposes when you create your own reports and dashboards.

Run a report

This section demonstrates how to manually submit a CloudBees Analytics report to CloudBees Analytics engine and to the Elasticsearch database. It assumes you have deployment data in your database.

Define the report

Use the CloudBees Analytics UI to define search criteria.

  • Select Analytics Reports: the reports list displays.

  • Select the Build outcome report, this would be a custom report you previously created. The Report editor displays the report in advanced mode and assumes you have deployment data in your database.

  • Select Tabular preview to run the report and see the result set.

At this point, you can toggle between the Report editor to tweak the report and Tabular preview to see the results. Click Save when you are satisfied with the report criteria.

To change a report bundled with CloudBees Analytics, you must first copy it and change the copy.

Issuing CloudBees Analytics API requests

Use the runReports CloudBees Analytics API command to run a report. The example below uses ectool to issue the command that runs the Build outcome report.

ectool runReport "CloudBees" "Build outcome"

Results similar to these below are returned in the response:

<response requestId="1" nodeId="192.168.5.138">
    <result>
      <deployment_outcome_count>26</deployment_outcome_count>
      <total>26</total>
      <deployment_outcome>success</deployment_outcome>
      <avg_deployment_duration>530769.23</avg_deployment_duration>
    </result>
  </response>

CloudBees Analytics API commands are also available from the CloudBees Software Delivery Automation Swagger UI available via https://<cloudbees-cd-server_hostname>/rest/doc/v1.0/.

  • For complete information about runReport and all CloudBees Analytics API commands, see CloudBees Analytics.

  • For additional information about creating and using CloudBees Analytics reports, see Creating a Report .

Issuing Elasticsearch API requests

This section shows you how to issue Elasticsearch API requests to the database via cURL.

The Elasticsearch version shipped with CloudBees Analytics version 10.2 has been updated from v6.6.2 to v7.10.2. As such, this update may create breaking changes in your custom reports. All changes related to the new version are described in the Elasticsearch documentation. The following change may effect your reports the most. * Accessing missing document values will throw an error. The doc['field'].value will throw an exception if the document is missing a value for the field field.

+ To check if a document is missing a value, you can use doc['field'].size() == 0.

Prerequisites

  • Elasticsearch database login credentials. Defaults are:

    • user: reportinguser

    • password: changeme

  • A tool for issuing REST POST requests. For example, cURL

  • A tool for Base64 encoding

Accessing the Elasticsearch database

From a web browser, access the Elasticsearch database with the URL https://<hostname>:9200, where <hostname> is the same hostname as your CloudBees Analytics server. Alternatively, navigate to the Administration ConfigurationsAnalytics server to determine your Elasticsearch database URL.

Enter your credentials for Elasticsearch in the sign-in dialog; confirm you get a response similar to this:

{
  "name" : "<hostname>",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "nbV6hwg0QM-ICIIGcvK2AQ",
  "version" : {
    "number" : "6.6.2",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "3bd3e59",
    "build_date" : "2019-03-06T15:16:26.864148Z",
    "build_snapshot" : false,
    "lucene_version" : "7.6.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Base64 encoded credentials

Your credentials are required to be in a Base64 encoded format. To generate this encoded credential string you can use `ectool ` or an on line tool, the latter is not recommended for production use since you cannot be assured you data is held privately by an unknown party.

ectool evalDsl '"reportinguser:changeme".bytes.encodeBase64().toString()'
<response requestId="1" nodeId="10.1.1.101">
    <value>cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ==</value>
</response>
The resulting key is `cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ`.

Create the request

The following cURL request queries the ef-build table in the database. In the context of the URL below, ef-build-* tells Elasticsearch to return data from the CloudBees Analytics ( ef- ) deployment table. The asterisk is a wildcard indicating to return deployment data for all time; the indices are tagged with the year (-2021).

curl -X POST \
 'https://<hostname>:9200/ef-build-*/_search?pretty=' \
 -H 'Authorization: Basic cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ ' \
 -H 'Content-Type: application/json' \
 -d '{"size": 10,"query": {"bool": {}}}'

[source,json]

Assuming the CloudBees Analytics server instance has deployment data, running this request should return something like the following (only the first few lines are shown here). Report data is returned in _source elements in the result set. Refer to schemas documented in Understanding the analytics data model for description of result set fields. In this case, Build.

{
  {
    "took" : 8,
    "timed_out" : false,
    "_shards" : {
      "total" : 2,
      "successful" : 2,
      "skipped" : 0,
      "failed" : 0
    },
    "hits" : {
      "total" : 1624,
      "max_score" : 1.0,
      "hits" : [
        {
          "_index" : "ef-build-2021",
          "_type" : "doc",
          "_id" : "5CVyPXcBLrhx_rZ26wl7",
          "_score" : 1.0,
          "_source" : {
            "timestamp" : "2021-01-26T06:47:55.751Z",
            "reportObjectType" : "build",
            "controllerName" : "inst2-test_9a715976-cf41-43c8-9429-94b0099523ce",
            "startTime" : "2021-01-26T06:46:55.751Z",
            "duration" : 60000,
            "buildStatus" : "SUCCESS",
    ...

Search the database

The following cURL request queries the ef-build-* table in the database searching for records whose buildStatus field is set to SUCCESS. In the context of the URL below, ef-build-* tells Elasticsearch to return data from the CloudBees Analytics (ef-) deployment table. The asterisk is a wildcard indicating to return deployment data for all time; the indices are tagged with the year (-2021).

curl -X POST \
 'https://<hostname>:9200/ef-deployment%2A/_search?q=deploymentOutcome:success' \
 -H 'Authorization: Basic cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ ' \
 -H 'Content-Type: application/json'

Assuming the Electric CloudBees Software Delivery Automation instance has been used for deployments, running this request should return something like the following (only the first few lines are shown here). The field in the request is highlighted in blue.

{
  {
    "took" : 8,
    "timed_out" : false,
    "_shards" : {
      "total" : 2,
      "successful" : 2,
      "skipped" : 0,
      "failed" : 0
    },
    "hits" : {
      "total" : 1624,
      "max_score" : 1.0,
      "hits" : [
        {
          "_index" : "ef-build-2021",
          "_type" : "doc",
          "_id" : "5CVyPXcBLrhx_rZ26wl7",
          "_score" : 1.0,
          "_source" : {
            "timestamp" : "2021-01-26T06:47:55.751Z",
            "reportObjectType" : "build",
            "controllerName" : "inst2-test_9a715976-cf41-43c8-9429-94b0099523ce",
            "startTime" : "2021-01-26T06:46:55.751Z",
            "duration" : 60000,
            "buildStatus" : "SUCCESS",
    ...

Elasticsearch keywords

Since report definitions are constructed using Elasticsearch DSL queries behind the scenes, some knowledge of Elasticsearch is valuable to build custom reports. Extensive documentation and examples reside online on the Elasticsearch website https://www.elastic.co/. Elasticsearch training is beyond the scope of this document.