Elasticsearch Basics

The Elasticsearch database stores all Flow, Plugin, and user-contributed data for use by DevOps Insight reports and dashboards. While developing reports and dashboards, it is useful to see what data is available in the Elasticsearch database. Information in this section helps you to set up and examine data and to run reports.

Examining Data in the Elasticsearch Database

This section helps you to set up and directly query the Elasticsearch database. It can be useful to examine report data and to run reports directly against the Elasticsearch database for debugging purposes when you create your own reports and dashboards.

Run a Report

This section demonstrates how to manually submit a DevOps Insight report to the Elasticsearch database. It assumes you have deployment data in your database.

Using the CloudBees Flow UI

  • From the main CloudBees Flow main menu, select Reports from the DevOps Analytics section: the reports list displays.

  • Select the AvgDeploymentDurations report. The Report Editor displays the report in advanced mode. assumes you have deployment data in your database.

  • Select Tabular Preview to run the report and see the result set.

At this point, you can toggle between the Report Editor to tweak the report and Tabular Preview to see the results. Click Save when you are satisfied with your report.

To change a report bundled with DevOps Insight, you must first copy it and change the copy.

Using the CloudBees Flow API

Use the runReports CloudBees Flow API command to run a report. The example below uses ectool to issue the command that runs the AvgDeploymentDurations report.

ectool runReport "Electric Cloud" "AvgDeploymentDurations"

Results similar to these below are returned in the response:

<response requestId="1" nodeId="192.168.5.138">
    <result>
      <deployment_outcome_count>26</deployment_outcome_count>
      <total>26</total>
      <deployment_outcome>success</deployment_outcome>
      <avg_deployment_duration>530769.23</avg_deployment_duration>
    </result>
  </response>

CloudBees Flow API commands are also available from the CloudBees Flow Swagger UI available via https://<CloudBees_Flow_server_hostname>/rest/doc/v1.0/.

  • For complete information about runReport and all DevOps Insight API commands, see DevOps Insight.

  • For additional information about creating and using DevOps Insight reports, see Creating a Report .

Issuing Elasticsearch API Requests

This section shows you how to issue Elasticsearch API requests to the database via cURL.

Prerequisites

  • Elasticsearch database login credentials. Defaults are:

    • user: reportinguser

    • password: changeme

  • A tool for issuing REST POST requests. For example, cURL

  • A tool for Base64 encoding

Accessing the Elasticsearch Database

From a web browser, access the Elasticsearch database with the URL https://<hostname>:9200, where <hostname> is the same hostname as your Electric Flow server. Alternatively, navigate to the Administration > DevOps Insight Server page via the Platform Home page link on the CloudBees Flow main menu to determine your Elasticsearch database URL.

Enter your credentials for Elasticsearch in the sign-in dialog; confirm you get a response similar to this:

{
  "name" : "<hostname>",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "nbV6hwg0QM-ICIIGcvK2AQ",
  "version" : {
    "number" : "6.6.2",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "3bd3e59",
    "build_date" : "2019-03-06T15:16:26.864148Z",
    "build_snapshot" : false,
    "lucene_version" : "7.6.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Base64 Encoded Credentials

Your credentials are required to be in a Base64 encoded format. To generate this encoded credential string you can use `ectool ` or an on line tool, the latter is not recommended for production use since you cannot be assured you data is held privately by an unknown party.

ectool evalDsl '"reportinguser:changeme".bytes.encodeBase64().toString()'
<response requestId="1" nodeId="10.1.1.101">
    <value>cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ==</value>
  </response>

The resulting key is cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ.

Create the Request

The following cURL request queries the ` ef-deployments` table in the database. In the context of the URL below, ef-deployment* tells Elasticsearch to return data from the Electric Flow ( ef- ) deployment table. The asterisk is a wildcard indicating to return deployment data for all time; the indices are tagged with the year (-2019).

curl -X POST \
 'https://<hostname>:9200/ef-deployment%2A/_search?pretty=' \
 -H 'Authorization: Basic cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ ' \
 -H 'Content-Type: application/json' \
 -d '{"size": 10,"query": {"bool": {}}}'

Assuming the Electric Flow instance has been used for deployments, running this request should return something like the following (only the first few lines are shown here).

{
   "took":40,
   "timed_out":false,
   "_shards":{
      "total":2,
      "successful":2,
      "skipped":0,
      "failed":0
   },
   "hits":{
      "total":1106,
      "max_score":0.26351982,
      "hits":[
         {
            "_index":"ef-deployment-2019",
            "_type":"doc",
            "_id":"scCnGW0BIgjCcfPASuWA",
            "_score":0.26351982,
            "_source":{
               "pipelineName":"Demo pipeline",
               "manual":false,
               "taskName":"App task",
               "waitForPlannedStart":false,
               "processStepId":"f1c18fec-d38b-11e9-bf23-0050568f29b0",
               "applicationName":"DEMO App",
               "jobStepStart":"2019-09-10T05:30:56.339Z",
               "environmentId":"ea29e603-d38b-11e9-bf23-0050568f29b0",
               "flowRuntimeName":"DEMO Release_Demo pipeline_6_20190909223009",
               "jobStatus":"running",
               "reportObjectType":"deployment",
               "projectName":"Platform Job Details DEMO Project",
               "environmentName":"DEMO Env",
               "archived":false,
               "taskType":"PROCESS",
               "jobId":"1e1a4156-d38c-11e9-9818-0050568f29b0",
               "launchedByUser":"admin",
    ...

Search the Database

The following cURL request queries the ` ef-deployments` table in the database searching for records whose deploymentOutcome ` field is set to `success. In the context of the URL below, ef-deployment* tells Elasticsearch to return data from the Electric Flow ( ef- ) deployment table. The asterisk is a wildcard indicating to return deployment data for all time; the indices are tagged with the year (-2019).

curl -X POST \
 'https://<hostname>:9200/ef-deployment%2A/_search?q=deploymentOutcome:success' \
 -H 'Authorization: Basic cmVwb3J0aW5ndXNlcjpjaGFuZ2VtZQ ' \
 -H 'Content-Type: application/json'

Assuming the Electric Flow instance has been used for deployments, running this request should return something like the following (only the first few lines are shown here). The field in the request is highlighted in blue.

{
   "took":6,
   "timed_out":false,
   "_shards":{
      "total":2,
      "successful":2,
      "skipped":0,
      "failed":0
   },
   "hits":{
      "total":1106,
      "max_score":0.26351982,
      "hits":[
         {
            "_index":"ef-deployment-2019",
            "_type":"doc",
            "_id":"scCnGW0BIgjCcfPASuWA",
            "_score":0.26351982,
            "_source":{
               "pipelineName":"Demo pipeline",
               "manual":false,
        ...
               "flowRuntimeStateOutcome":"success",
               "applicationProjectName":"Platform Job Details DEMO Project",
               "stageName":"Demo Stage",
               "releaseProjectName":"Platform Job Details DEMO Project",
               "jobStepType":"manual",
               "jobName":"12_Deploy App Process_DEMO App_Platform Job Details DEMO Project_20190909223035",
               "processStepName":"Manual component step",
             "deploymentOutcome":"success",
        ...
            }
         },
       ...

Elasticsearch Keywords

Since report definitions are constructed using Elasticsearch DSL queries behind the scenes, some knowledge of Elasticsearch is valuable to build custom reports. Extensive documentation and examples reside online on the Elasticsearch website (https://www.elastic.co/). Elasticsearch training is beyond the scope of this document.

Copyright © 2010-2020 CloudBees, Inc.Online version published by CloudBees, Inc. under the Creative Commons Attribution-ShareAlike 4.0 license.CloudBees and CloudBees DevOptics are registered trademarks and CloudBees Core, CloudBees Flow, CloudBees Flow Deploy, CloudBees Flow DevOps Insight, CloudBees Flow DevOps Foresight, CloudBees Flow Release, CloudBees Accelerator, CloudBees Accelerator ElectricInsight, CloudBees Accelerator Electric Make, CloudBees CodeShip, CloudBees Jenkins Enterprise, CloudBees Jenkins Platform, CloudBees Jenkins Operations Center, and DEV@cloud are trademarks of CloudBees, Inc. Most CloudBees products are commonly referred to by their short names — Accelerator, Automation Platform, Flow, Deploy, Foresight, Release, Insight, and eMake — throughout various types of CloudBees product-specific documentation. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Jenkins is a registered trademark of the non-profit Software in the Public Interest organization. Used with permission. See here for more info about the Jenkins project. The registered trademark Jenkins® is used pursuant to a sublicense from the Jenkins project and Software in the Public Interest, Inc. Read more at www.cloudbees.com/jenkins/about. Apache, Apache Ant, Apache Maven, Ant and Maven are trademarks of The Apache Software Foundation. Used with permission. No endorsement by The Apache Software Foundation is implied by the use of these marks.Other names may be trademarks of their respective owners. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this content, and CloudBees was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this content, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.