Elasticsearch Basics

4 minute read

The Elasticsearch database stores all Flow, Plugin, and user-contributed data for use by DevOps Insight reports and dashboards. While developing reports and dashboards, it is useful to see what data is available in the Elasticsearch database. Information in this section helps you to set up and examine data and to run reports.

Examining Data in the Elasticsearch Database

This section helps you to set up and directly query the Elasticsearch database. It can be useful to examine report data and to run reports directly against the Elasticsearch database for debugging purposes when you create your own reports and dashboards.

Run a Report

This section demonstrates how to manually submit a DevOps Insight report to the Elasticsearch database. It assumes you have deployment data in your database.

Using the CloudBees Flow UI

  • From the main CloudBees Flow main menu, select Reports from the DevOps Analytics section: the reports list displays.

  • Select the AvgDeploymentDurations report. The Report Editor displays the report in advanced mode. assumes you have deployment data in your database.

  • Select Tabular Preview to run the report and see the result set.

At this point, you can toggle between the Report Editor to tweak the report and Tabular Preview to see the results. Click Save when you are satisfied with your report.

To change a report bundled with DevOps Insight, you must first copy it and change the copy.

Using the CloudBees Flow API

Use the runReports CloudBees Flow API command to run a report. The example below uses ectool to issue the command that runs the AvgDeploymentDurations report.

ectool runReport "Electric Cloud" "AvgDeploymentDurations"

Results similar to these below are returned in the response:

<response requestId="1" nodeId="192.168.5.138"> <result> <deployment_outcome_count>26</deployment_outcome_count> <total>26</total> <deployment_outcome>success</deployment_outcome> <avg_deployment_duration>530769.23</avg_deployment_duration> </result> </response>

CloudBees Flow API commands are also available from the CloudBees Flow Swagger UI available via https://<cloudbees-flow-server_hostname>/rest/doc/v1.0/.

  • For complete information about runReport and all DevOps Insight API commands, see DevOps Insight.

  • For additional information about creating and using DevOps Insight reports, see Creating a Report .

Issuing Elasticsearch API Requests

This section shows you how to issue Elasticsearch API requests to the database via cURL.

Prerequisites

  • Elasticsearch database login credentials. Defaults are:

    • user: reportinguser

    • password: changeme

  • A tool for issuing REST POST requests. For example, cURL

  • A tool for Base64 encoding

Accessing the Elasticsearch Database

From a web browser, access the Elasticsearch database with the URL https://<hostname>:9200, where <hostname> is the same hostname as your Electric Flow server. Alternatively, navigate to the Administration > DevOps Insight Server page via the Platform Home page link on the CloudBees Flow main menu to determine your Elasticsearch database URL.

Enter your credentials for Elasticsearch in the sign-in dialog; confirm you get a response similar to this:

{ "name" : "<hostname>", "cluster_name" : "elasticsearch", "cluster_uuid" : "nbV6hwg0QM-ICIIGcvK2AQ", "version" : { "number" : "6.6.2", "build_flavor" : "oss", "build_type" : "tar", "build_hash" : "3bd3e59", "build_date" : "2019-03-06T15:16:26.864148Z", "build_snapshot" : false, "lucene_version" : "7.6.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }

Base64 Encoded Credentials

Your credentials are required to be in a Base64 encoded format. To generate this encoded credential string you can use `ectool ` or an on line tool, the latter is not recommended for production use since you cannot be assured you data is held privately by an unknown party.

ectool evalDsl '"reportinguser:changeme".bytes.encodeBase64().toString()'
<response requestId="1" nodeId="10.1.1.101"> <value>cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ==</value> </response> [source,txt]

The resulting key is cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ.

Create the Request

The following cURL request queries the ` ef-deployments` table in the database. In the context of the URL below, ef-deployment* tells Elasticsearch to return data from the Electric Flow ( ef- ) deployment table. The asterisk is a wildcard indicating to return deployment data for all time; the indices are tagged with the year (-2019).

curl -X POST \ 'https://<hostname>:9200/ef-deployment%2A/_search?pretty=' \ -H 'Authorization: Basic cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ ' \ -H 'Content-Type: application/json' \ -d '{"size": 10,"query": {"bool": {}}}' [source,txt]

Assuming the Electric Flow instance has been used for deployments, running this request should return something like the following (only the first few lines are shown here).

{ "took":40, "timed_out":false, "_shards":{ "total":2, "successful":2, "skipped":0, "failed":0 }, "hits":{ "total":1106, "max_score":0.26351982, "hits":[ { "_index":"ef-deployment-2019", "_type":"doc", "_id":"scCnGW0BIgjCcfPASuWA", "_score":0.26351982, "_source":{ "pipelineName":"Demo pipeline", "manual":false, "taskName":"App task", "waitForPlannedStart":false, "processStepId":"f1c18fec-d38b-11e9-bf23-0050568f29b0", "applicationName":"DEMO App", "jobStepStart":"2019-09-10T05:30:56.339Z", "environmentId":"ea29e603-d38b-11e9-bf23-0050568f29b0", "flowRuntimeName":"DEMO Release_Demo pipeline_6_20190909223009", "jobStatus":"running", "reportObjectType":"deployment", "projectName":"Platform Job Details DEMO Project", "environmentName":"DEMO Env", "archived":false, "taskType":"PROCESS", "jobId":"1e1a4156-d38c-11e9-9818-0050568f29b0", "launchedByUser":"admin", ...

Search the Database

The following cURL request queries the ` ef-deployments` table in the database searching for records whose deploymentOutcome ` field is set to `success. In the context of the URL below, ef-deployment* tells Elasticsearch to return data from the Electric Flow ( ef- ) deployment table. The asterisk is a wildcard indicating to return deployment data for all time; the indices are tagged with the year (-2019).

curl -X POST \ 'https://<hostname>:9200/ef-deployment%2A/_search?q=deploymentOutcome:success' \ -H 'Authorization: Basic cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ ' \ -H 'Content-Type: application/json' [source,txt]

Assuming the Electric Flow instance has been used for deployments, running this request should return something like the following (only the first few lines are shown here). The field in the request is highlighted in blue.

{ "took":6, "timed_out":false, "_shards":{ "total":2, "successful":2, "skipped":0, "failed":0 }, "hits":{ "total":1106, "max_score":0.26351982, "hits":[ { "_index":"ef-deployment-2019", "_type":"doc", "_id":"scCnGW0BIgjCcfPASuWA", "_score":0.26351982, "_source":{ "pipelineName":"Demo pipeline", "manual":false, ... "flowRuntimeStateOutcome":"success", "applicationProjectName":"Platform Job Details DEMO Project", "stageName":"Demo Stage", "releaseProjectName":"Platform Job Details DEMO Project", "jobStepType":"manual", "jobName":"12_Deploy App Process_DEMO App_Platform Job Details DEMO Project_20190909223035", "processStepName":"Manual component step", "deploymentOutcome":"success", ... } }, ...

Elasticsearch Keywords

Since report definitions are constructed using Elasticsearch DSL queries behind the scenes, some knowledge of Elasticsearch is valuable to build custom reports. Extensive documentation and examples reside online on the Elasticsearch website https://www.elastic.co/. Elasticsearch training is beyond the scope of this document.