How to set up a Load Balancer for HA?

Issue

How to set up a Load Balancer for CJOC and CC HA

Environment

CloudBees Jenkins Operations Center (CJOC)
Client controller (CC)
Load Balancer

Resolution

The load balancer needs to be configured to load balance HTTP(S) traffic to the CloudBees Jenkins Instances which form the HA singleton.
The Load Balancer needs to perform a health check on the Jenkins Instances on /ha/health-check. Configure the load balancer to perform a health check by issuing a HEAD HTTP request on the nodes on /ha/health-check. The primary node is the only one giving you HTTP 200 response code.
The communication through JNLP port should not go through the load Balancer unless your company policies obligates you to configure JNLP through the load balancer.We should bypass the load balancer by adding the Java Argument in each OC instance -Dhudson.TcpSlaveAgentListener.hostName=<OC_MACHINE_HOSTNAME>. If you don’t want to perform a restart, after adding the Java Argument you can execute TcpSlaveAgentListener.CLI_HOST_NAME="CJOC_HOSTNAME" in Manage Jenkins -> Script Console in the Operations Center machine.

To check if the set-up was correctly performed you can run the command below and check that the header X-Jenkins-CLI-Host is present with the value we added.

$curl -I https://cje-1.jenkins.example.com:8080/
HTTP/1.1 200 OK
Date: Thu, 13 Jul 2017 11:49:29 GMT
X-Content-Type-Options: nosniff
Set-Cookie: JSESSIONID.7b40bd3d=hbc0fd3lhh0w1bg9cusqtea8r;Path=/;HttpOnly
Expires: 0
Cache-Control: no-cache,no-store,must-revalidate
X-Hudson-Theme: default
Content-Type: text/html;charset=UTF-8
X-Hudson: 1.395
X-Jenkins: 2.60.1.1
X-Jenkins-Session: 1af0d556
X-Hudson-CLI-Port: 54789
X-Jenkins-CLI-Port: 54789
X-Jenkins-CLI2-Port: 54789
X-Jenkins-CLI-Host: cje-1.jenkins.example.com
X-Frame-Options: sameorigin
X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAlx9/b60USmo2Eexm8TBtkcG881tkAP77iBbct0gDlSLaDdhbP7thD86Mu3WQVxmq23Rzqd/WR2VpN+JaQCQYttQ/KdxqnkTPZFO4vYy7n9E0pWgw8VKVZ9WEPfwfDEf4tHm/NmT4i43YrG4oeJvrVrGpwDp7vBviZNzQqOlcxLsr6njOX3+aZMYGkSdkduKgtKB5tV5qJSQxNFw/3I9HuxSZlaWMyni9NhAs1jGJ7EaxKT3Jtix56EY+vIRY43YhLU6YvnU/O6R2H9KdmxyFjthr1H3x8Wjfo1qYqePIL73bjA5guRyZZirbKL+1ep9LjTqJZYpH3EUjGoiBtYeyiwIDAQAB
Content-Length: 17904
Server: Jetty(9.2.z-SNAPSHOT)

Route HTTP(S) Traffic

Configure the "frontend" of the Load Balancer to route incoming HTTP(S) requests to the two CloudBees Jenkins Instances "alpha" and "beta" in the "backend".

The combination of Hostname / VIP / Port must be unique for each of the cluster: that is, if the port is fixed (443 usually if using HTTPS) the VIP can be shared in the different clusters if the hostnames are different. The VIP is used to select the correct "backend" based on the Host header.

You can run the following script from Manage Jenkins->Script Console to check if headers are set properly:

   println Jenkins.instance.rootUrlFromRequest
   println org.kohsuke.stapler.Stapler.currentRequest.getHeader('X-Forwarded-For')
   println org.kohsuke.stapler.Stapler.currentRequest.getHeader('X-Forwarded-Host')
   println org.kohsuke.stapler.Stapler.currentRequest.getHeader('X-Forwarded-Proto')
   println org.kohsuke.stapler.Stapler.currentRequest.getHeader('X-Forwarded-Port')
   println org.kohsuke.stapler.Stapler.currentRequest.serverPort
   println org.kohsuke.stapler.Stapler.currentRequest.serverName

Route JNLP Traffic

As it was previously specified, CloudBees does not recommend to route the JNLP traffic unless it is a must coming from your company policies - or because OC and CCs are in different networks only reachable between them when there is a load balancer in the middle.

In case this is necessary, you need to ensure that the idle timeout is bigger than the PingThread timeout. By default, the PingThread timeout is 4 minutes, so a minimum timeout of 300 seconds is recommended. If the default PingThread was modified, then you need to modify the timeout in the load balancer accordingly.

AWS ELB

In AWS there in a idle timeout that you can configure under Description -> Attributes -> Idle timeout. Ensure that this is configured at very least with 300 seconds.

Finally, ensure that under the Listeners section the JNLP port is configured to forward the packages using the protocol TCP.

F5 considerations: OneConnection

If you are using a feature called OneConnection or something similar to improve the networking performance, please deactivate it for this environment as we have the experience it makes the connection to break.