Creating a CloudBees Analytics server cluster

5 minute readScalability

The CloudBees Analytics server uses the Elasticsearch search engine to gather data from the CloudBees CD/RO server for use in the CloudBees Analytics dashboards such as the Deployments, Releases, and Release Command Center dashboards. Elasticsearch is a powerful search and analysis engine, and part of this power lies in the ability to scale for better performance and stability.

An Elasticsearch cluster is a collection of one or more nodes (servers) that holds your entire data and provides federated indexing and search capabilities across all nodes. A cluster is identified by a unique name, which is set using the Elasticsearch Cluster name provided to the CloudBees Analytics server installer. This name is important, because a node can be part of a cluster only if the node is set up to join the cluster by its name.

A node is a single server that is part of your cluster, stores your data, and participates in the cluster’s indexing and search capabilities. Just like a cluster, a node is identified by a name, which is set using the Elasticsearch Node name provided to the CloudBees Analytics server installer.

A cluster can have any number of nodes. At least three nodes are recommended for a cluster.

Creating a typical CloudBees Analytics cluster

To set up a typical CloudBees Analytics cluster deployment, complete the following steps.

Choosing the security mode for the cluster

If the mode will be password protected, then you must choose a password. This password must be used during installation for every node in cluster and it must be the same for all nodes. If the cluster will not be password protected, then all nodes must be installed using the same mode.

Installing the first node in the cluster

Use one of the following installer modes:

  • GUI mode—In the Cluster Settings installer screen, check the This is the first node in the cluster checkbox.

    For details about this screen and when it appears during the installation session, see Running an Advanced Graphical User Interface Installation .

  • Console mode—Answer Yes at the Do you want to specify additional Elasticsearch cluster mode settings? prompt and at the Is this node the first node to be installed in the Elasticsearch cluster? prompt.

    For details about when these prompts appear during the installation session, see Running an Advanced Command-Line Installation .

  • Silent mode—Additional arguments are not needed (cluster mode is the default).

This node will be a master node and a data node automatically.

Retrieve the certificate file containing a CA-signed certificate for the CloudBees Analytics server, which is located at <DATA_DIR>/conf/reporting/elasticsearch/signing-ca.p12. This file is needed for installing the CloudBees Analytics server on all other cluster nodes.

Installing each additional node

Perform this step on each additional node. These settings should be the same across all additional nodes. Use one of the following installer modes:

  • GUI mode—In the Cluster Settings pane, uncheck the This is the first node in the cluster checkbox. For details about this screen and when it appears during the installation session, see Running an Advanced Graphical User Interface Installation .

    Then specify the certificate file containing a CA-signed certificate retrieved from the first node on the Advanced Settings pane.

  • Console mode—Answer Yes at the Do you want to specify additional Elasticsearch cluster mode settings? prompt and No at the Is this node the first node to be installed in the Elasticsearch cluster? prompt.

    For details about when these prompts appear during the installation session, see Running an Advanced Command-Line Installation .

    Then specify the certificate file containing a CA-signed certificate retrieved from the first node at the PKCS#12 file containing a CA-signed certificate for the CloudBees Analytics Server prompt.

    You can leave this entry blank for a new installation in non-clustered mode or for the first node in clustered mode. In this case, the installer will generate a new self-signed certificate and will use it to sign other TLS certificates.
  • Silent mode—Use the --elasticsearchNodeAdditional command line argument and specify the certificate file containing a CA-signed certificate retrieved from the first node using the --elasticsearchCACertificateFile command line argument.

The cluster name must be the same for all nodes.
During node installation, the list of other nodes should be specified. It is mandatory for additional nodes and optional for the first node. You should specify all available master nodes in this list.

Configuring the load balancer and the CloudBees CD/RO server

You must configure a load balancer to balance one port for the CloudBees Analytics services:

  • TCP port of the Elasticsearch service. The default port number is 9200.

You must configure the CloudBees CD/RO server to use the load balancer host by using the instructions in Creating a CloudBees Analytics Server Cluster.

DNS issue and publish host setting

The Publish host setting is critical for nodes in a cluster. By default, this setting contains the value of the node’s hostname. The cluster will not work if this hostname resolves as an internal IP address that is unreachable by other nodes or as the IP address of the loopback interface (such as 127.0.0.1). If this is the case, then messages similar to the following will be in the log file:

[2018-08-30T09:49:23,907][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node0] no known master node, scheduling a retry [2018-08-30T09:49:25,449][INFO ][o.e.d.z.ZenDiscovery ] [node0] failed to send join request to master [{node1}{-xG7eVHnRbGvGVgAYwoukA}{nshut08WQ9iRoPwpJomomA}{node1}{127.0.1.1:9300}], reason [RemoteTransportException[[node0][127.0.1.1:9300][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node0}{rVpbjN3VQyGv3mf-FG7bOA}{--V1HU-qToeAMkxPENGeNw}{node0}{127.0.1.1:9300}] not master for join request]; ], tried [3] times

To fix this issue, you must specify a hostname that resolves to a real IP address by setting --elasticsearchPublishHost in console or silent install modes or in the corresponding field in GUI installation mode. Also, the exact IP address can be passed to this setting.

Ensuring a healthy cluster before upgrade or reconfiguration operations

The upgrade or reconfiguration processes require that the settings be written to the cluster data, but this is impossible if the cluster is unhealthy. In this case, the installation pauses and waits for a healthy cluster. Therefore, you should upgrade every node separately and one by one.

Make sure that the cluster has green or yellow status before and after the upgrade or reconfiguration of each node. To do so, use the curl utility:

$ curl -k 'https://reportuser:<YOUR PASSWORD>@localhost:9200/_cluster/health?pretty' { "cluster_name" : "elasticsearch", "status" : "green", "timed_out" : false, "number_of_nodes" : 5, "number_of_data_nodes" : 4, "active_primary_shards" : 1, "active_shards" : 1, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }

If the cluster is unhealthy, then the command returns the following result:

$ curl -k 'https://reportuser:<YOUR_PASSWORD>@localhost:9200/_cluster/health?pretty' Search Guard not initialized (SG11). See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md

At first, you should upgrade all master nodes. Then you can start the upgrade on the other nodes.

The exception is changing the cluster name setting. When this reconfiguration is performed over the first nodes, then the cluster cannot be formed, because other nodes have a different cluster name. In this case, the upgrade will pause at the “Starting services” stage. At this point, you can start the upgrades for the other master nodes. When the number of upgraded nodes equals or exceeds the minimum number of master-eligible nodes, then the cluster will be formed, and all paused upgrades will complete successfully.

Upgrading from version 8.4 and earlier

When you generate the signing CA certificate file, all certificates are regenerated during the upgrade from CloudBees Analytics version 8.4 and earlier to version 8.5 and later. The old certificates are saved with the *.backup extension in the configuration directories.

If you used custom certificates, you can restore them by renaming .pem.backup and .jks.backup in the DATA_DIR/conf/reporting/elasticsearch directory. You must restart the CloudBees Analytics services when you rename certificate files.

Changing the password for secure access to a CloudBees Analytics cluster

To change the password for secure access to a CloudBees Analytics cluster, you must perform the reconfiguration with the corresponding changes over each node in the cluster.