Install HA (active/active) on CloudBees CI on traditional platforms

High Availability (HA) on Windows controllers is not supported.

This document provides specific instructions to install and configure High Availability HA for CloudBees CI on traditional platforms. Instructions and standard methods to install CloudBees CI on traditional platforms can be found in this section.

Run client controllers in HA mode

CloudBees CI on traditional platforms controllers use a built-in HTTP(S) server (Jetty) when run as a self-contained application directly (java -jar *.war) or indirectly via native package managers or Docker containers. Refer to Install client controllers for more information. This built-in server is compatible with HA.

Client controllers in HA) mode will not work when using WAR files on external web servlet containers like Apache Tomcat.

Standalone controllers (not connected to any operations center) can run in HA mode.

Load balancer

You need to set up a load balancer to route HTTP and, if desired, WebSocket, traffic to all the replicas.

There are a lot of different load balancers to choose from. CloudBees CI users must configure and set up the load balancer according to their specific requirements.

The load balancer needs two configurations applied:

The load balancer must be configured with sticky sessions.
You should set up a health check against /whoAmI/api/json?tree=authenticated

Storage

All replicas must point to the same $JENKINS_HOME location.

For CloudBees CI on traditional platforms, you must use a shared file system, such as NFS.

To properly configure the NFS client on your controller instances, follow the instructions from the NFS Guide knowledge base article.

Controller Service Configurations

Whether you installed CloudBees CI on traditional platforms using a package manager, that is yum, dnf, zypper or apt, there are still more changes you need to make to the service configuration files on each replica.

If you installed via yum, dnf, or zypper, that is, an RPM install, you need to update:

/etc/sysconfig/cloudbees-core-cm

If you installed via apt, that is, a DEB install, you need to update:

/etc/default/cloudbees-core-cm

The following sections document what the values should be updated in the respective service configuration files.

`JENKINS_HOME`

This should be changed from /var/lib/cloudbees-core-cm to whatever the NFS mount is on the replica.

For instance, if you created the NFS mount as /mnt/nfs_cc_home, then the value changes from:

JENKINS_HOME="/var/lib/cloudbees-core-cm"

to:

JENKINS_HOME="/mnt/nfs_cc_home"

`JENKINS_ARGS`

There is one mandatory argument to set in JENKINS_ARGS and one optional argument.

First, add an argument to a directory on local disk (not shared disk) for --pluginroot. This specifies where the plugin files should be extracted.

If you did an RPM install, the value should be:

--pluginroot=/var/cache/cloudbees-core-cm/plugins▼

The plugins subdirectory will be automatically created when the service starts.

If you did a DEB install, the value should be:

--pluginroot=/var/cache/$NAME/plugins▼

The optional argument to set is --prefix. You will set this argument if you are placing your controller under a domain name instead of using a subdomain.

For example, let’s assume that you want the URL to the controller to be:

https://cloudbees-ci.example.com/cc1/▼

In this case, you would set the argument to:

--prefix=/cc1▼

The value is the same for both RPM and DEB installs.

For a complete example that includes both arguments in addition to the default values for a DEB install looks like:

JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT --pluginroot=/var/cache/$NAME/plugins --prefix=/cc1"▼

Java options

Java 11 is required to run CloudBees CI on traditional platforms 2.414.2.2 or higher

The Java options and system properties discussed below are only the values for configuring High Availability.

For the overall JVM recommended arguments to run CloudBees CI on traditional platforms, please review the JVM Recommended Arguments section of Prepare Jenkins for Support.

The following Java options and system properties are required for the controllers to run in HA mode:

--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED
--add-modules=java.se
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.management/sun.management=ALL-UNNAMED
--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
-Djenkins.model.Jenkins.crumbIssuerProxyCompatibility=true
-DexecutableWar.jetty.disableCustomSessionIdCookieName=true
-Dcom.cloudbees.jenkins.ha=false
-Dcom.cloudbees.jenkins.replication.warhead.ReplicationServletListener.enabled=true▼

If you did an RPM install, ensure the variable is initialized as an array JENKINS_JAVA_OPTIONS=(…), not a string JENKINS_JAVA_OPTIONS="…", then you can add the following:

JENKINS_JAVA_OPTIONS+=("--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED")
JENKINS_JAVA_OPTIONS+=("--add-modules=java.se")
JENKINS_JAVA_OPTIONS+=("--add-opens=java.base/java.lang=ALL-UNNAMED")
JENKINS_JAVA_OPTIONS+=("--add-opens=java.base/sun.nio.ch=ALL-UNNAMED")
JENKINS_JAVA_OPTIONS+=("--add-opens=java.management/sun.management=ALL-UNNAMED")
JENKINS_JAVA_OPTIONS+=("--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED")
JENKINS_JAVA_OPTIONS+=("-Djenkins.model.Jenkins.crumbIssuerProxyCompatibility=true")
JENKINS_JAVA_OPTIONS+=("-DexecutableWar.jetty.disableCustomSessionIdCookieName=true")
JENKINS_JAVA_OPTIONS+=("-Dcom.cloudbees.jenkins.ha=false")
JENKINS_JAVA_OPTIONS+=("-Dcom.cloudbees.jenkins.replication.warhead.ReplicationServletListener.enabled=true")▼

If you did a DEB install, you will update the default JAVA_ARGS value from:

JAVA_ARGS="-Djava.awt.headless=true"▼

to:

JAVA_ARGS="-Djava.awt.headless=true --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-modules=java.se --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Djenkins.model.Jenkins.crumbIssuerProxyCompatibility=true -DexecutableWar.jetty.disableCustomSessionIdCookieName=true -Dcom.cloudbees.jenkins.ha=false -Dcom.cloudbees.jenkins.replication.warhead.ReplicationServletListener.enabled=true"▼

Controllers running in HA mode automatically configure cache-related properties to use per-replica temporary folder locations. Folders containing these caches are located outside the shared $JENKINS_HOME folder to avoid conflicts.

These folders are used by:

The GitHub branch source, to minimize REST API calls.
Generic Git repositories to store the repository cache.
Both multibranch Pipeline and standalone Pipeline using the script from SCM option, when using plain Git repository references, in certain configurations.
Pipeline Groovy libraries when not used in clone mode (with or without caching mode enabled), to maintain checkouts of library SCM repositories.

You may use the CACHE_DIR environment variable to specify a custom base directory for caches. This is useful when you need to use a location different from the default /tmp folder, such as when the new location offers better performance for caching operations.

This base directory must have the correct permissions and ownership, which must be set manually. Here’s a basic example of how to create the directory:

sudo mkdir -p /var/cache/cloudbees-ci
sudo chmod 700 /var/cache/cloudbees-ci
sudo chown -R cloudbees-core-cm:cloudbees-core-cm /var/cache/cloudbees-ci▼

The subdirectories will be created automatically after the base directory is created.

Network requirements

Controller replicas must be able to connect each other to keep the replicas in sync and with a consistent behavior. The following ports are used, unless changed, for that communication:

HTTP port (8080)
Hazelcast port (5701)
Inbound TCP port for agents (if used) (50000)

Refer to Configuring network requirements for further information.

Hazelcast configuration on CloudBees CI on traditional platforms

Hazelcast is a key element in the HA architecture.

In standard CloudBees CI on traditional platforms installation:

A discovery process uses files in $JENKINS_HOME to form the HA cluster with controller replicas.
Some Hazelcast configuration parameters can be managed through JVM arguments when starting the controller. Port switching is an example.

For more information, refer to the HA troubleshooting section.

For advanced use cases with special network configurations, admins can override the default Hazelcast configuration by providing a custom XML or YAML file.

Setting the hazelcast.config system property to the path of the custom configuration file enables the Controllers to use this custom file to configure Hazelcast.

When using a custom configuration file:

The discovery process to create the cluster is not performed.
Every Hazelcast configuration parameter must be manually set in the custom configuration using the hazelcast.config system property or in other applicable properties.

Refer to Hazelcast documentation for more information regarding creating a custom configuration file.

Setup wizard

When customizing the list of plugins to install, be sure to add the CloudBees High Availability (Active/Active) plugin. Alternatively, use Configuration as Code (CasC) to define the controller, and include the plugin cloudbees-replication.

Do not include the CloudBees High Availability (Active/Passive) Management plugin (cloudbees-ha) that is used by the older active-passive HA system.

If the cloudbees-ha plugin is installed, it should be uninstalled and the replicas should be restarted prior to installing the cloudbees-replication plugin.

Restarting controllers

If you need to restart a controller, for example, after installing a plugin, make sure that every replica is restarted.

Additional resources

Migrating from HA(active/passive) to HA on CloudBees CI on traditional platforms

Getting started with HA on traditional platforms