HA (active/active) considerations

3 minute read

Troubleshooting HA

CloudBees CI High Availability provides tools to troubleshoot possible problems on controller replicas running in HA mode. To enable them, navigate to Manage Jenkins  CloudBees CI High Availability.

  • To enable HA Developer Mode, select Status from the left navigation pane. When enabled, a controller running in developer mode provides additional information, like the replica used by the user, or the replica executing a build.

  • Select HA Script Console to run scripts across all the controller replicas, and gather information from all of them.

Nodes and agents

CloudBees supports a range of agent connection modes in High Availability, but each agent must have only one executor. As an agent can connect to only a single replica at a time, agents with multiple executors cannot be properly shared and are not supported by CloudBees CI High Availability.

You can share a high-capacity computer among several concurrent builds, if desired, by connecting multiple agents to the replicated controller.

Ensure that you use a unique remote root directory so each agent has its own workspace.

Shared libraries

To use Groovy libraries, CloudBees recommends that you set them to a new “clone” mode and configure Git to use a shallow clone.

For concurrent access, if you update library checkouts in a common directory (such as $JENKINS_HOME/workspace/) or use the caching system, it can cause problems. An administrative monitor guides you to make these changes.

Non-Pipeline projects

Problems can arise in High Availability mode with non-Pipeline project types, such as Freestyle, Matrix, or Maven. When these project types are run, other replicas can load completed build metadata, but cannot take over the build successfully. As a result, if a replica terminates, any builds running on it are immediately aborted.

Horizontal scalability

The benefit of having multiple replicas should always be balanced against the associated cost according to your business case. Scaling horizontally with many replicas will have diminishing returns as the number of replicas increases.

Plugin installation and HA

Plugins can be managed and installed from the Manage Jenkins  Plugins screen. When using HA with multiple replicas, dynamic loading of plugins (plugin installation without restarting CloudBees CI) is not supported. Therefore, you must restart each replica of the controller to install or upgrade plugins.

Dynamic loading of plugins not supported
Figure 1. Dynamic loading of plugins not supported

In a CloudBees CI on modern cloud platforms with a managed controller running in HA mode, when selecting Restart Jenkins when installation is complete and no jobs are running, a rolling restart is performed, and when completed, new plugin versions are available in all replicas.

In a {CC-TRAD} running in HA mode with multiple replicas, you must restart all controller replicas either manually or using your own automation.

When the controller is running in HA mode with only one replica, the behaviour is the same as a non-HA controller.

HA and REST-API endpoints

When running a controller in HA mode, any API pull-based endpoint only returns information about the controller replica that responds to the API request.

Examples of these endpoints are:

  • The /metrics endpoint of the Metrics plugin.

  • The /monitoring endpoint provided by the Monitoring plugin.

In this situation, inaccurate information may be provided in the response, depending on the metrics of requested information.

For example, if making a HTTP API query for JVM heap usage, the returned value would only correspond to the replica that processed the request, and not provide any insight into other replicas. However, other information, like the number of projects, is accurate because it is automatically synchronized among all the controller replicas.

Stage View and Blue Ocean plugins

The Pipeline: Stage View and Blue Ocean plugins may not accurately display running builds that are owned by other replicas.

CloudBees recommends the CloudBees Pipeline Explorer plugin.