Customer Environment Validation Tests for a CJOC environment

Article ID:219982128
5 minute readKnowledge base

Issue

  • Customer Environment Validation Tests for a CJOC environment

Environment

  • CloudBees Jenkins Operations Center (CJOC)

  • CloudBees Jenkins Enterprise (CJE)

Resolution

This document provides a mechanism to validate customers’s environments before running CloudBees Jenkins Operations Center. The KB proposes a number of tests that should be carried out to assert the underlying infrastructure works as intended.

Introduction: Connection flow between client controllers and the OC instance

A simple connection flow looks like the image below:

cjoc connection flow
  1. Client controller sends HEAD request to CJOC root URL

  2. Client controller parses HEAD response headers looking for X-Jenkins and X-Jenkins-CLI-Port.

  3. Client controller initiates a TCP connection to the port specified in X-Jenkins-CLI-Port against the host specified in X-Jenkins-CLI-Host (or the host from the CJOC root URL if the X-Jenkins-CLI-Host header is absent)

If from CJE instance under https://<CJE_URL>/script we launch the script below we can see those headers exposed by CJOC.

def url = new URL("https://<CJOC_URL>/");
def connection = url.openConnection();
println("Response Headers");
println("================");
for (def e in connection.getHeaderFields()) {
  println("${e.key}: ${e.value}");
}
println("\nResponse status: HTTP/${connection.responseCode}\n");

An output like the one below is expected:

Response Headers
================
null: [HTTP/1.1 200 OK]
X-Jenkins: [1.625.16.2 (CloudBees Jenkins Operations Center 1.8)]
X-Hudson: [1.395]
Content-Length: [16451]
Expires: [0]
Set-Cookie: [JSESSIONID.d35196d9=x9jsjuq6ugw3x5t38e6ecfjt;Path=/;HttpOnly]
X-Jenkins-CLI-Port: [6587]
X-Hudson-CLI-Port: [6587]
X-SSH-Endpoint: [127.0.0.1:49691]
Server: [Jetty(winstone-2.8)]
X-Content-Type-Options: [nosniff]
Cache-Control: [no-cache,no-store,must-revalidate]
X-Frame-Options: [sameorigin]
X-Hudson-Theme: [default]
X-Instance-Identity: [MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApZe+Wd2zoAiWalHu/wgZaFzJS56XYxv1XejB/ml8k4tb+AutXvUhDvxlRYvmkp7CV/NvF3e2YfgH5WLnt5IXXxn5a3FsDQ3xDtN/+kv4+jwTqbZnOORtJB0xET7Pak7fpjrTkUpsR8lGhHdHVf2iQxHRJoeHFblYajNHPx9U7zkrXgLA4xUqqwCPmUys/QXAcVEl6IL/1jmxtdLFQb38NNPr6CDfhrH7DNkwvTX6EDmn1Zljw+AsFBmDbfm/uoPqUCNrrQk/OvIrKebABPdJL0YoIY7naehd8ezNiICrf92TkZNopCrTUNuq6kpe8rhHW0oQf5o0KSrSi869nXG8FwIDAQAB]
X-Jenkins-Session: [5e3b53cb]
Content-Type: [text/html;charset=UTF-8]
X-Jenkins-CLI2-Port: [6587]

Response status: HTTP/200

In case you get Response status: HTTP/403 (Forbidden) is due to the CJOC Security Option is enabled and it is also expected.

After the explanation above, we should be able to understand how the flow looks like when pushing the connection details from CJOC to CJE

cjoc push connection details
cjoc client controller registration sequence
  1. Browser POST to JOC

  2. JOC redirect via browser to CJE

  3. CJE HTTP HEAD to JOC

  4. CJE extracts X-Jenkins-CLI-Port

  5. CJE establish remoting protocol to CJOC through TCP on JNLP port exposed by CJOC

Summary:

  1. Communication between CJOC and CJE happens at HTTP level - on the in which CJOC is accessible through the browser - and at TCP level through the JNLP port exposed by CJOC

  2. Communication is ALWAYS initiated by CJE trying to connect with CJOC using HTTP and TCP protocols

Customer CJOC Configuration Validation

On this section, we need to ensure that we have set-up everything correctly at Jenkins level. The image below shows a typical customer environment. A similar environment (plus a firewall configuration) can be found on this tutorial for demo purposes.

cjoc typical customer environment

In both CJOC and CJE level we need to ensure that:

  1. The Jenkins URL was set-up under Manage Jenkins -> Configure System

  2. Jenkins is exposing a JNLP under Manage Jenkins -> Configure Global Security. The best practice is to set-up a fixed port in both CJOC and CJE instances, but this port should be different in CJOC and CJE Note: Use a different JNLP port for each of the CJE instances connected to the CJOC.

[Support] Customer CJOC Configuration Validation

If you are going to open a support ticket because you cannot figure out why CJE cannot connect to CJOC, then ensure that you provide us a table similar to the one below so we can understand how your environment looks like.

Node Type IP Address DNS Name HTTP(S) PORT JNLP Port $JENKINS_HOME

CJOCa

CJOC primary

joc-a.acme.com

CJOCb

CJOC backup

joc-b.acme.com

CJOC

CJOC VIP

jenkins.acme.com/operations-center

controller1a

controller1 primary

je1-a.acme.com

controller1b

controller1 backup

je1-b.acme.com

controller1

controller1 VIP

jenkins.acme.com/je1

CJOC/CJE backup are available for High Availability (HA) set-up environments

Customer CJOC Networking Validation

On this section we will continue with the same typical customer’s architecture we used on the section before:

cjoc typical customer environment

At networking level CJE needs to be able to access CJOC through HTTP - on the port in which CJOC is accessible- and TCP on the JNLP port exposed by CJOC.

CJOC connectivity tests

Please, provide Test ID and Pass/Fail results for the following table:

Test ID Details Notes

cjoc-desktop-http

From desktop: browser access to HTTP(s) port on <CJOC VIP>.
Expected: Jenkins home page to display

As shown on the above figure: jenkins.acme.com/operations-center

cjoc-desktop-http-primary-instance

From desktop: browser access HTTP(s) port on <CJOC primary>.
Expected: Jenkins home page to display

As shown on the above figure: joc-a.acme.com

cjoc-desktop-http-backup-instance

From desktop: browser access HTTP(s) port on <CJOC backup>.
Expected: "This node is standing by in case the primary node fails over" message

As shown on the above figure: joc-b.acme.com

cjoc-dns-resolution

From desktop: ping <CJOC_VIP>, ping <joc-a>, ping <joc-b> - check resolved IPs as per table above

cjoc-http-tcp-vip

From desktop:
a) curl -I -v <CJOC_VIP>:<HTTP_PORT>
Obtain value from X-Jenkins-CLI-Port
b) telnet <CJOC_VIP> <CLI PORT>
Expected:
Trying ::1... Connected to localhost Escape character is ...

cjoc-http-tcp-primary-instance

Shutdown CJOC-b and log on CJOC-a machine, and then:
a) curl -I -v <CJOC_VIP>:<HTTP_PORT>
Obtain value from X-Jenkins-CLI-Port
b) telnet <CJOC_VIP> <CLI PORT>
Expected:
Trying ::1... Connected to localhost Escape character is ...

cjoc-http-tcp-backup-instance

Shutdown CJOC-a and log on CJOC-b machine, and then:
a) curl -I -v <CJOC_VIP>:<HTTP_PORT>
Obtain value from X-Jenkins-CLI-Port
b) telnet <CJOC_VIP> <CLI PORT>
Expected:
Trying ::1... Connected to localhost Escape character is ...

controller connectivity tests

Please, provide Test ID and Pass/Fail results for the following table:

Test ID Details Notes

cje-desktop-http

From desktop: browser access to HTTP(s) port on <CJE VIP>.
Expected: Jenkins home page to display

As shown on the above figure: jenkins.acme.com/je1

cje-desktop-http-primary-instance

From desktop: browser access HTTP(s) port on <controller1 primary>.
Expected: Jenkins home page to display

As shown on the above figure: je1-a.acme.com

cje-desktop-http-backup-instance

From desktop: browser access HTTP(s) port on <controller1 backup>.
Expected: "This node is standing by in case the primary node fails over" message

As shown on the above figure: je1-b.acme.com

cje-http-tcp-primary-instance

Shutdown CJEa and log on CJEb machine, and then:
a) curl -I -v <CJOC_VIP>:<HTTP_PORT>
Obtain value from X-Jenkins-CLI-Port
b) telnet <CJOC_VIP> <CLI PORT>
Expected:
Trying ::1... Connected to localhost Escape character is ...

cje-http-tcp-backup-instance

Shutdown CJEb and log on CJEa machine, and then:
a) curl -I -v <CJOC_VIP>:<HTTP_PORT>
Obtain value from X-Jenkins-CLI-Port
b) telnet <CJOC_VIP> <CLI PORT>
Expected:
Trying ::1... Connected to localhost Escape character is ...

[Support] Customer CJOC Networking Validation

In case you plan to open a support ticket, please ensure that all these tests are done before opening a ticket and provide the output on a .txt so we can check that the outputs are the expected.

JENKINS_HOME tests

Ssh to each server and cat the $JENKINS_HOME/config.xml.

CJE validation for non-fresh instances

Typical scenarios are:

  1. CJE instance that was never managed by any CJOC and now moves to a CJE-CJOC architecture.

  2. CJE instance that was previous managed by one CJOC and now migrates to another CJOC instance (CJE Migration between CJOC instances)

CJE Admin Login validation (Optional)

Just in the case that a user with Administer permission is not able to login into the CJE instance to connect to a CJOC, CJE Security needs to be disable by following this instructions. For instance, when CJE Jenkins Dashboard ($CJE_URL) redirects to registration page ($CJE_URL/registration).

Disable previous CJOC connection

Before connecting a CJE to a CJOC, the former CJOC Connector in the CJE configurations needs to be disabled.

Go to $CJE_URL/configure > Operations Center Connector > Disable

customerEnvValTestCJOC CJE disablePriorConnect