Deploy CloudBees CI across multiple Kubernetes clusters

Modern Cloud Platforms

This content applies only to CloudBees CI on modern cloud platforms.

CloudBees CI on modern cloud platforms can be provisioned and managed across multiple Kubernetes clusters and cloud providers. You can run operations center on a main Kubernetes cluster that manages controllers across other Kubernetes clusters. These clusters can be hosted on all the major cloud providers. You can set up a multicluster configuration to provision controllers in different clusters located in the same cloud provider or a different one.

You can provision, update, and manage CloudBees CI controllers on multiple Kubernetes clusters from a common operations center, which allows users to:

Separate divisions or business units who need to run workloads in individual clusters.
Leverage Helm charts and Configuration as Code to configure, provision, and update using version-controlled configuration files.

Common use cases for multicluster

Controller in multiple Kubernetes clusters

In this use case, the operations center runs in one Kubernetes cluster. Some controllers run in the same Kubernetes cluster as the operations center. Any number of additional controllers run in a separate Kubernetes cluster.

Each instance, both operations center and each controller, is tied to its own Kubernetes cluster and cannot move after it is provisioned.

Refer to Provisioning managed controllers in multiple Kubernetes clusters.
Agents in separate Kubernetes clusters

In this use case, agents are deployed in a separate Kubernetes cluster from the Kubernetes cluster where a controller is set up.

The controller can be configured to schedule agents between several Kubernetes clouds. Note that no control mechanism is available to spread the load across different Kubernetes clusters because a pod template is tied to a Kubernetes cloud.

Refer to Provisioning agents in a separate Kubernetes cluster from a managed controller.

By default, managed controllers are created in the same namespace as the operations center instance. You can, however, create a managed controller in a separate namespace or OpenShift project.

Network considerations for multicluster

Keep in mind the following information about networking for provisioned controllers and agents:

When provisioning a controller on a remote Kubernetes cluster, the Kubernetes API must be accessible from the operations center.

Upon startup, a controller connects back to the operations center through HTTP or HTTPS when using Websockets. If you are not using Websockets, it connects using inbound TCP. If the controller is located in a different Kubernetes cluster:
- The operations center must provide its external hostname so that the controller can connect back.
- If using WebSockets, the operations center HTTP / HTTPS port must be exposed.
- If not using WebSockets, the operations center inbound TCP port must be exposed.
Refer to Adding external client controllers and Using WebSockets to connect controllers to operations center.
When provisioning an agent on a remote Kubernetes cluster, the Kubernetes API must be accessible from the managed controller.

Upon startup, an agent connects back to its controller. If the agent is in a different Kubernetes cluster:
- The controller must provide its external hostname so that the controller can connect back.
- If using WebSockets, the controller HTTP / HTTPS port must be exposed.
- If not using WebSockets, the controller inbound TCP port must be exposed.
Refer to Connecting inbound agents.
In order for controllers to be accessible, each Kubernetes cluster must have its own Ingress controller installed, and the configured endpoint for each controller should be consistent with the DNS pointing to the Ingress controller of that Kubernetes cluster.
Bandwidth, latency, and reliability can be issues when running components in different regions or clouds.
- Operations center <> Controller connection is required for features like shared agents, licensing (every 72 hours) or SSO. An unstable connection can result in the inability to sign in, unavailable shared agents or an invalid license if the disconnected time is too long.
- Controller <> agent connection is required to transfer commands and logs. An unstable connection can result in failed builds, unexpected delays, or timeouts while running builds.

TLS for remote controllers in a multicluster environment

The recommended approach to set up Ingress TLS for controllers in remote clusters is to add the secret to the namespace where the controller is deployed, and then add the TLS to each remote controller’s Ingress (through the YAML field).

Remove controllers from a cluster

If you remove a controller from the operations center, the controller’s Persistent Volume Claim, which represents the JENKINS_HOME, is left behind.

To remove a controller from a cluster:

Type the following command:
```
kubectl delete pvc <pvc-name>▼
```
The <pvc-name> is always jenkins-home-<controller name>-0.

Troubleshoot multicluster and multicloud

What should I do if my controller cannot connect back to the operations center?: Use kubectl to retrieve the controller. Then, inspect where it tries to connect and how it matches the configuration that is defined in operations center.
My controller starts and shows as connected in operations center. Why isn’t my controller browsable?: This can be caused by either a DNS/Ingress misconfiguration or a URL misconfiguration.