Installation for CloudBees CI on traditional platforms (active/active)

High Availability (active/active) on Windows controllers is not supported.

This document covers specific instructions to install and configure High Availability (active/active) for CloudBees CI on traditional platforms. Instructions and standard methods to install CloudBees CI on traditional platforms can be found in this section.

Run client controllers in High Availability (active/active) mode

CloudBees CI on traditional platforms controllers default to run a built-in Jetty server when run as an app (java -jar *.war). However, controllers can also run from a package manager installation or as Docker containers. Refer to Install client controllers for more information.

Client controllers in High Availability (active/active) mode will not work when using WAR files on external web servlet containers like Apache Tomcat.

Standalone controllers (not connected to any operations center) can run in High Availability (active/active) mode.

Load balancer

You need to set up a load balancer to route HTTP and, if desired, WebSocket, traffic to all the replicas.

There are a lot of different load balancers to choose from, so it’s up to you to figure out how to set up your specific one.

The load balancer needs two configurations applied:

The load balancer must be configured with sticky sessions.
You should setup a health check against /whoAmI/api/json?tree=authenticated

Storage

All replicas must point to the same $JENKINS_HOME location.

For CloudBees CI on traditional platforms, you must use a NFS compatible shared file system.

To properly configure the NFS client on your controller instances, follow the instructions from the NFS Guide knowledge base article.

Controller Service Configurations

Whether you installed CloudBees CI on traditional platforms using a package manager, that is yum, dnf, zypper or apt, there are still more changes you need to make to the service configuration files on each replica.

If you installed via yum, dnf, or zypper, that is, a RPM install, you need to update:

/etc/sysconfig/cloudbees-core-cm

If you installed via apt, that is, a DEB install, you need to update:

/etc/default/cloudbees-core-cm

The following sections document what the values should be updated in the respective service configuration files.

`JENKINS_HOME`

This should be changed from /var/lib/cloudbees-core-cm to whatever the NFS mount is on the replica.

For instance, if you created the NFS mount as /mnt/nfs_cc_home, then the value changes from:

JENKINS_HOME="/var/lib/cloudbees-core-cm"

to:

JENKINS_HOME="/mnt/nfs_cc_home"

`JENKINS_ARGS`

There is one mandatory argument to set in JENKINS_ARGS and one optional argument.

First, add an argument to a directory on local disk (not shared disk) for --pluginroot. This specifies where the plugin files should be extracted.

If you did an RPM install, the value should be:

--pluginroot=/var/cache/cloudbees-core-cm/plugins▼

The plugins subdirectory will be automatically created when the service starts.

If you did a DEB install, the value should be:

--pluginroot=/var/cache/$NAME/plugins▼

The optional argument to set is --prefix. You will set this argument if you are placing your controller under a domain name instead of using a subdomain.

For example, let’s assume that you want the URL to the controller to be:

https://cloudbees-ci.example.com/cc1/▼

In this case, you would set the argument to:

--prefix=/cc1▼

The value is the same for both RPM and DEB installs.

For a complete example that includes both arguments in addition to the default values for a DEB install looks like:

JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT --pluginroot=/var/cache/$NAME/plugins --prefix=/cc1"▼

Java options

Java 11 is required to run CloudBees CI on traditional platforms 2.414.2.2 or higher

The Java options and system properties discussed below are only the values for configuring High Availability.

For the overall JVM recommended arguments to run CloudBees CI on traditional platforms, please review the JVM Recommended Arguments section of Prepare Jenkins for Support.

The following Java options and system properties are required for the controllers to run in HA mode:

--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED
--add-modules=java.se
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.management/sun.management=ALL-UNNAMED
--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
-Djenkins.model.Jenkins.crumbIssuerProxyCompatibility=true
-DexecutableWar.jetty.disableCustomSessionIdCookieName=true
-Dcom.cloudbees.jenkins.ha=false
-Dcom.cloudbees.jenkins.replication.warhead.ReplicationServletListener.enabled=true
-Djenkins.plugins.git.AbstractGitSCMSource.cacheRootDir=/var/cache/cloudbees-core-cm/caches/git
-Dorg.jenkinsci.plugins.github_branch_source.GitHubSCMSource.cacheRootDir=/var/cache/cloudbees-core-cm/caches/github-branch-source▼

If you did a RPM install, ensure the variable is initialized as an array JENKINS_JAVA_OPTIONS=(…), not a string JENKINS_JAVA_OPTIONS="…", then you can add the following:

JENKINS_JAVA_OPTIONS+=("--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED")
JENKINS_JAVA_OPTIONS+=("--add-modules=java.se")
JENKINS_JAVA_OPTIONS+=("--add-opens=java.base/java.lang=ALL-UNNAMED")
JENKINS_JAVA_OPTIONS+=("--add-opens=java.base/sun.nio.ch=ALL-UNNAMED")
JENKINS_JAVA_OPTIONS+=("--add-opens=java.management/sun.management=ALL-UNNAMED")
JENKINS_JAVA_OPTIONS+=("--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED")
JENKINS_JAVA_OPTIONS+=("-Djenkins.model.Jenkins.crumbIssuerProxyCompatibility=true")
JENKINS_JAVA_OPTIONS+=("-DexecutableWar.jetty.disableCustomSessionIdCookieName=true")
JENKINS_JAVA_OPTIONS+=("-Dcom.cloudbees.jenkins.ha=false")
JENKINS_JAVA_OPTIONS+=("-Dcom.cloudbees.jenkins.replication.warhead.ReplicationServletListener.enabled=true")
JENKINS_JAVA_OPTIONS+=("-Djenkins.plugins.git.AbstractGitSCMSource.cacheRootDir=/var/cache/cloudbees-core-cm/caches/git")
JENKINS_JAVA_OPTIONS+=("-Dorg.jenkinsci.plugins.github_branch_source.GitHubSCMSource.cacheRootDir=/var/cache/cloudbees-core-cm/caches/github-branch-source")▼

If you did a DEB install, you will update the default JAVA_ARGS value from:

JAVA_ARGS="-Djava.awt.headless=true"▼

to:

JAVA_ARGS="-Djava.awt.headless=true --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-modules=java.se --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Djenkins.model.Jenkins.crumbIssuerProxyCompatibility=true -DexecutableWar.jetty.disableCustomSessionIdCookieName=true -Dcom.cloudbees.jenkins.ha=false -Dcom.cloudbees.jenkins.replication.warhead.ReplicationServletListener.enabled=true -Djenkins.plugins.git.AbstractGitSCMSource.cacheRootDir=/var/cache/$NAME/caches/git -Dorg.jenkinsci.plugins.github_branch_source.GitHubSCMSource.cacheRootDir=/var/cache/$NAME/caches/github-branch-source"▼

Note the last two Git related items in both examples. The base directory, that is, /var/cache/cloudbees-core-cm/caches or /var/cache/$NAME/caches, should be a disk local to the replica and not on a shared disk.

However, this base directory will need to be created by you and have the correct permissions and ownership set. Here’s a basic example of how to create the directory:

sudo mkdir -p /var/cache/cloudbees-core-cm/caches
sudo chmod 700 /var/cache/cloudbees-core-cm/caches
sudo chown -R cloudbees-core-cm:cloudbees-core-cm /var/cache/cloudbees-core-cm/caches▼

The subdirectories will be created automatically after the base directory is created.

Network requirements

Controller replicas must be able to connect each other to keep the replicas in sync and with a consistent behavior. The following ports are used, unless changed, for that communication:

HTTP port (8080)
Hazelcast port (5701)
Inbound TCP port for agents (if used) (50000)

Refer to Configuring network requirements for further information.

Hazelcast configuration on CloudBees CI on traditional platforms

Hazelcast is a key element in the High Availability (active/active) (HA) architecture.

In standard CloudBees CI on traditional platforms installation:

A discovery process uses files in $JENKINS_HOME to form the HA cluster with controller replicas.
Some Hazelcast configuration parameters can be managed through JVM arguments when starting the controller. Port switching is an example.

For more information, refer to the High Availability (active/active) troubleshooting section.

For advanced use cases with special network configurations, admins can override the default Hazelcast configuration by providing a custom XML or YAML file.

Setting the hazelcast.config system property to the path of the custom configuration file enables the Controllers to use this custom file to configure Hazelcast.

When using a custom configuration file:

The discovery process to create the cluster is not performed.
Every Hazelcast configuration parameter must be manually set in the custom configuration using the hazelcast.config system property or in other applicable properties.

Refer to Hazelcast documentation for more information regarding creating a custom configuration file.

Setup wizard

When customizing the list of plugins to install, be sure to add the CloudBees High Availability (Active/Active) plugin. Alternatively, use Configuration as Code (CasC) to define the controller, and include the plugin cloudbees-replication.

Do not include the CloudBees High Availability (Active/Passive) Management plugin (cloudbees-ha) that is used by the older active-passive HA system.

If the cloudbees-ha plugin is installed, it should be uninstalled and the replicas should be restarted prior to installing the cloudbees-replication plugin.

Restarting controllers

If you need to restart a controller, for example, after installing a plugin, make sure that every replica is restarted.

Additional resources

Migrating from High Availability(active/passive) to High Availability(active/active) on CloudBees CI on traditional platforms

Getting started with High Availability on traditional platforms