Get ready for HA (active/active)

Once you understand what HA is, what it provides, its benefits and the main elements within the HA architecture, this section presents a recommended workflow for deploying an HA cluster.

CloudBees recommends the following steps to get your cluster ready for HA:

Review your network and storage configurations to meet the requirements for HA.
Install the HA controllers.
Configure your agents to work with HA.
Audit your jobs to ensure they are compatible with HA.
Review additional valuable information about HA.

Review your network and storage configurations to meet the requirements for HA

Network configuration for HA

As described in HA fundamentals, controllers running in HA mode require controller replicas to communicate with each other. When using CloudBees CI on traditional platforms, your network configuration must open the following required ports to allow this communication:

Hazelcast nodes installed in the replicas. The default Hazelcast port is 5701.
A reverse-proxy HTTP or TCP connection to access resources from one replica that belong or are connected to another replica. Resources might include:
- Running builds.
- WebSocket inbound agents.
- TCP inbound agents from outside the CloudBees CI network.

For CloudBees CI on traditional platforms, the load balancer used to distribute the traffic among the replicas must be configured to use sticky sessions. There are no additional considerations needed for CloudBees CI on modern cloud platforms.

If an HTTP proxy configuration is used, replica IPs and hostnames must be added to the non-proxy list.

Install a storage system that meets the requirements for HA

In CloudBees CI on modern cloud platforms, HA requires consistent shared storage between replicas. PersistentVolumeClaims (PVC) used for this purpose must accept an access mode of ReadWriteMany. The underlying storage provided depends on the Kubernetes platform used. Refer to Install HA on CloudBees CI on modern cloud platforms for more information.

In CloudBees CI on traditional platforms, as all the replicas share the same JENKINS_HOME directory. They must be able to access this shared directory, which is typically mounted on the same path for all the replicas. You must use a shared file system like NFS to store the JENKINS_HOME directory. Refer to Install HA on CloudBees CI on traditional platforms for more information.

Migrate your non-HA controller to HA

CloudBees CI controllers not running in HA mode can be migrated to HA mode. Migrated controllers must meet the specific storage requirements described before.

If the managed controller already uses a volume with ReadWriteMany capabilities, the migration process only requires switching the accessMode of the PersistentVolumeClaim (PVC) used. There is no need to move data in this scenario.

Migrating an existing managed controller to HA describes the process to migrate a non-HA controller to an HA controller for CloudBees CI on modern cloud platforms.

For CloudBees CI on traditional platforms, no migration is required if the client controller is already using a shared file system for JENKINS_HOME. If the client controller is not using a shared file system, and assuming that the destination volume is already mounted and is shared file system, the following steps provide a general overview of the migration process:

Set the $JENKINS_HOME environment variable to the destination volume, the NFS-compatible shared filesystem used by the controller replicas when running in HA mode.
Perform an initial sync between the current $JENKINS_HOME volume and the destination volume. Use, for example, rsync as described below:
```
rsync -avvu --delete <old-JENKINS_HOME> <new-JENKINS_HOME>(1)▼
```
1 Replace <old-JENKINS_HOME> with the path for the previous $JENKINS_HOME, and <new-JENKINS_HOME> with the destination path used as the new $JENKINS_HOME.
Stop the non-HA controller.
Perform a new sync between the current $JENKINS_HOME volume and the destination volume.
```
rsync -avvu --delete <old-JENKINS_HOME> <new-JENKINS_HOME>(1)▼
```
1 Replace <old-JENKINS_HOME> with the path for the previous $JENKINS_HOME, and <new-JENKINS_HOME> with the destination path used as the new $JENKINS_HOME.
Apply the configuration change that HA requires. Refer to Install HA (active/active) on CloudBees CI on traditional platforms for more information about these configuration changes.
Restart the controller, now running in HA mode.

Install the HA controllers

HA is available for CloudBees CI on modern cloud platforms and CloudBees CI on traditional platforms. Refer to Install HA (active/active) on CloudBees CI on modern cloud platforms and Install HA (active/active) on CloudBees CI on traditional platforms for more information.

High Availability (HA) on Windows controllers is not supported.

Configure your agents to work with HA

CloudBees CI supports a range of agent connection modes in HA, but each agent must have only one executor. As an agent can connect to only a single replica at a time, agents with multiple executors cannot be properly shared, and are not supported by CloudBees CI High Availability (HA).

A single executor requirement is not a problem for cloud agents as they normally define only one executor. They are provisioned, use this executor, and are de-provisioned later.

Permanent agents as multiple-executor agents for HA

Agents require one executor to work with a controller running in HA mode. However, you can configure permanent or static agents to automatically generate single-executor agent clones that simulate the behaviour of a multi-executor agent. To configure a permanent agent to simulate a multi-executor agent for HA, you can use one of the following methods:

Automatically from the GUI.
Manually from the GUI.
Using CloudBees Configurations as Code (CasC).

Automatically from the GUI.

If the permanent agent does not have one executor, CloudBees CI displays an administrative monitor. If this administrative monitor displays, and you select Apply Migration, CloudBees CI automatically migrates the permanent agent configuration to the required configuration for HA.

Figure 1. Multi-executor administrative monitor

Manually from the GUI

To manually configure your permanent agent to work with HA:

Navigate to the agent configuration screen.
Verify that the agent has only one executor.
In the Node Properties section, select the HA agent with multiple executor option, and set the number of executors according to your needs.

Figure 2. Number of executors for HA

For outbound agents (usually SSH agents), from the configuration screen, select CloudBees High Availability for the Availability (retention strategy) field. The agent is kept offline until requested by a replica. CloudBees CI, when needed, brings the permanent outbound agent online temporarily.

Figure 3. Availability for outbound agents

Using CloudBees Configurations as Code (CasC)

With CloudBees Configuration as Code (CasC) it is possible to configure a permanent agent as a multi-executor agent compatible with HA using code snippets similar to the examples below that display the non-HA compatible configuration and the changes requires to make it HA compatible:

nodes:
  - permanent:
      ...
      launcher:
        inbound: {}
      name: <your-permanent-agent-name>(1)
      numExecutors: 3(2)
      ...
      retentionStrategy: "always"▼

1	Replace `<your-permanent-agent-name>` with the name of your permanent agent.
2	Set the number of executors according to your needs.

nodes:
  - permanent:
      ...
      launcher:
        inbound: {}
      name: <your-permanent-agent-name>(1)
      numExecutors: 1(2)
      nodeProperties:
      - cloudbeesHighAvailabilityMultipleExecutors:(3)
          numExecutors: 3(4)
      ....
      retentionStrategy: "always"▼

1	Replace `<your-permanent-agent-name>` with the name of your permanent agent.
2	Set the non-HA option number of executors to `1`.
3	Use the `cloudbeesHighAvailabilityMultipleExecutors` property to define the permanent agent as a multi-executor agent for HA.
4	Set the number of HA compatible single-executor clones according to your needs.

nodes:
  - permanent:
      ...
      launcher:
        nioSsh:
        ...
      name: <your-permanent-agent-name>(1)
      numExecutors: 3(2)
      ...
      retentionStrategy: "demand"▼

1	Replace `<your-permanent-agent-name>` with the name of your permanent agent.
2	Set the number of executors according to your needs.

nodes:
  - permanent:
       ...
      launcher:
        nioSsh:
        ...
      name: <your-permanent-agent-name>(1)
      numExecutors: 1(2)
      nodeProperties:
      - cloudbeesHighAvailabilityMultipleExecutors:(3)
          numExecutors: 3(4)
      ....
      retentionStrategy: "cloudbeesHighAvailability"(5)▼

1	Replace `<your-permanent-agent-name>` with the name of your permanent agent.
2	Set the non-HA option number of executors to `1`.
3	Use the `cloudbeesHighAvailabilityMultipleExecutors` property to define the permanent agent as a multi-executor agent for HA.
4	Set the number of HA compatible single-executor clones according to your needs.
5	For outbound agents in HA controllers, set `retentionStrategy` always to `cloudbeesHighAvailability`.

When configured properly, the controller clones the original agent to create the executors. These read-only clones are managed by the controller itself, which adds or removes clones and reconfigures them in response to updates to the original permanent agent.

All the permanent agent clones have one executor and use different folders within the executors folder inside the original permanent agent root directory. Outbound agents automatically have the workDir settings for all the clones.

CloudBees CI displays the clones as individual one-executor agents in the Build executor status.

Agent clones providing muti-executor for HA and permanent agents

Figure 4. Agent clones providing multi-executor for HA and permanent agents

As permanent agent clones are read-only, users should avoid the Launch build agents via SSH (Non-blocking I/O) launch method. If this method is selected for a permanent agent connection, this connection fails. The Host Key Verification Strategy cannot use the Manually Trusted Key Verification Strategy as this strategy depends on permissions not granted to read-only clones.

Audit your jobs to ensure they are compatible with HA

Not every project or pipeline idiom is compatible with HA. When migrating your projects to HA, you must review the information in this section and rewrite your jobs or projects if necessary.

Pipeline projects are supported in HA mode.

Non-Pipeline projects like Freestyle, Matrix or Maven projects run in the replica where they started. Non-pipeline project builds are aborted when the replica running them ends for any reason, such as a rolling restart or upgrade, and no other replica can take over the build. Therefore, no HA is provided for those jobs.

Even though HA is not provided for non-pipeline projects, High Scalability (HS) can be achieved by adding more replicas to the cluster. However, if auto-scaling is enabled for CloudBees CI on modern cloud platforms, after a period of high workload and growth in the number of replicas, when the workload goes back to normal and the number of replicas decreases as a part of a scale-down process, non-Pipeline projects running in the removed replicas are aborted and not adopted by other replicas.

The Declarative Pipeline Migration Assistant can help you migrate your Freestyle or Maven projects to Declarative Pipeline projects.

The following Pipeline steps are not currently supported in HA controllers:

build
lock
milestone

If you plan to use controllers running in HA mode, you can define Using Pipeline Policies. Use these to either warn developers about those incompatible steps (Warning policy), or to prevent developers from using them (Fail policy). The rule used for this kind of policy is Pipeline idioms incompatible with Cloudbees High Availability (active/active), as displayed in the image below:

Figure 5. Pipeline policy rule for pipeline idioms or steps incompatible with HA

Refer to Using Pipeline Policies for more information about how to define and apply pipeline policies to your controller.

Review additional valuable information about HA

REST-API endpoints and HA

When running a controller in HA mode, requests to API pull-based endpoints may return information about the controller replica that responds to the API request instead of aggregated information about all the controller replicas part of the HA cluster. To retrieve aggregated information refer to the HA considerations page.

Blue Ocean and HA

The Blue Ocean plugin may not accurately display running builds that are owned by other replicas.

CloudBees recommends the CloudBees Pipeline Explorer plugin.

HA and costs

HA may increase the costs of running CloudBees CI as it requires additional resources, like those used to run the replicas and the required network filesystem.

Troubleshooting HA problems

Refer to High Availability (active/active) troubleshooting for information about troubleshooting problems with HA.

In addition to the general troubleshooting information, the CloudBees Knowledge Base contains articles about common issues and solutions for HA.