Manage controllers

When new teams join an organization, or existing teams start a new project, CloudBees CI makes it easy to provision a fully managed and access controlled controller per team. In CloudBees CI on modern cloud platforms, a controller is referred to as a managed controller.

Administrators can provision managed controllers from a standardized template, or they can allow team leads to provision their own managed controllers "on-demand." The number of controllers an environment can handle is limited only by the capacity of the cluster.

Upgrading managed controllers

To update a managed controller’s version, the administrator needs to:

Navigate to the managed controller Manage Jenkins Beekeeper Upgrade Assistant CAP Configuration.
Verify that Enroll this instance in the CloudBees Assurance Program is selected.
Verify that Allow automatic upgrades of plugins on restart is selected.
Select Save.
Navigate to Manage Jenkins Manage Plugins and under the Updates tab, select upgrade all plugins.
Restart the managed controller.
Update the version of the Docker image for the managed controller.
Once this version is updated and the managed controller is reprovisioned (this happens automatically via a rolling upgrade if using CloudBees High Availability (HA)), the managed controller and its default plugins will be upgraded to the latest versions defined in the image, while any non-default plugins will be left at their current versions.
Navigate to the managed controller Manage Jenkins Manage Plugins and under the Updates tab, select upgrade all plugins.
Restart the managed controller.

Bulk upgrades

To back up CloudBees CI on modern cloud platforms instances, CloudBees recommends that you use Kubernetes volume snapshots, Velero, or an equivalent commercial tool. This is especially useful for hibernated controllers because you can back up the volume offline. It provides a safer and more efficient method than online backups with cluster operations and the CloudBees Backup plugin.

When a cluster serves many teams and contains many controllers, an administrator can save time and greatly reduce the overhead of upgrading those controllers by creating a repeatable task to automate this process. This can be achieved by defining a cluster operation in operations center.

To create a task to perform a bulk upgrade:

Sign in to operations center as an administrator.
Create a New Item for Cluster Operations.
Select the managed controllers cluster operation
Select the pre-configured upgrade pattern that you want to use.
In the Uses Docker Image, pick a Docker image that is used by the cluster’s controllers as the upgrade target. Any controller in the cluster using the selected image will be affected by this cluster operation.
In the Steps section, select Backup Controller and configure the options as per Configure backups using the CloudBees Backup plugin.
In the Steps section, select Update Docker Image and then pick the new Docker to which you want to bulk-upgrade the targeted controllers.
Add a Reprovision step to restart the targeted controllers.

Once these settings are configured, the administrator can run the cluster operation to perform a bulk upgrade of their cluster’s controllers, or schedule the operation to run at a later time.

Optimize resources with hibernation of managed controllers

A common issue in installations with many managed controllers is having a cluster with a lot of managed controllers that are not being used most of the time. What typically happens in large organizations is multiple controllers are heavily used during working hours, and then sit idle during non-working hours. The idle controllers still cost the same even though they are not being used. Similarly static agents where the work has been completed can sit entirely unused and still occupy valuable infrastructure resources. Without the ability to load-balance overtaxed controllers and remove unused static agents, these resources can passively increase cloud spending and infrastructure resources. Large-scale organizations can end up paying significant amounts of money for idle and unused resources.

For those cases, CloudBees offers the managed controller hibernation feature, which helps you "turn off" idle or unused managed controllers.

For example, if you have managed controllers that are only being used by your development teams during business hours, you can reclaim those resources when they’re idle instead of having those managed controllers "up" around the clock.

Install and enable hibernation

To use hibernation:

Enable hibernation in the Helm chart.
Install the CloudBees managed controller Hibernation Plugin on any managed controller you want to be able to hibernate.
Enable the CloudBees managed controller Hibernation Plugin after installation.

Enable the hibernation monitor using Helm

Use the helm upgrade command to enable the managed controller Hibernation monitoring service in an existing Helm release of CloudBees CI.

If you have not already done so, create a custom values YAML file to store customizations for your instance. For instructions, refer to Using a custom YAML file to install CloudBees CI.
Add the following clause to your custom values file:
```
Hibernation:
  Enabled: true▼
```

Run Helm upgrade with the following command:

$ helm upgrade -f <name-of-custom-values-file>.yaml cloudbees-core <path-to-CB-helm-chart>▼

Enable hibernation in a separate namespace

If you are following the instructions for adding a specific Namespace with Helm section of “Managed controllers in specific Kubernetes namespaces” and you wish to enable hibernation for at least some controllers in that namespace, add the following to the values.yaml in those instructions.

OperationsCenter:
  Enabled: false
  HostName: cloudbees-core.example.com (1)
  Ingress:
    Class: nginx (2)
Hibernation:
  Enabled: true▼

1	Replace `cloudbees-core.example.com` with your domain name.
2	Replace `nginx` with your Ingress class name

A separate instance of the monitoring service is created in that namespace and handles controllers only in the same namespace. An ingress is also created in that namespace to make namespaced hibernation links accessible.

Since the generated ingresses will share a single host name, all API endpoints mentioned below support an infix ns/extra-namespace/ after hibernation/, which routes requests to the monitor running in a specific namespace.

Where appropriate, this infix is included automatically in URLs supplied by CloudBees CI:

A hibernated managed controller’s link in the operations center dashboard may look like: https://cloudbees-core.example.com/hibernation/ns/extra-namespace/redirect/your-controller/
The hook URL registered when creating a GitHub organization folder may look like: https://cloudbees-core.example.com/hibernation/ns/extra-namespace/queue/your-controller/github-webhook/

Include a secret in hibernation URLs

If the ingress used by CloudBees CI is exposed to a wide internal network, or even the Internet, you may want to protect the URLs used for hibernation. Otherwise, anyone who guesses the correct URL (predictable based on the controller URL) can maliciously force the controller to wake up and stay awake, even if they did not have permission to log in to the controller. To protect the monitor, add Hibernation.Protected to your values.yaml file:

Hibernation:
  Enabled: true
  Protected: true▼

A Kubernetes Secret named managed-master-hibernation-monitor-protected will be automatically generated. Any hibernation URLs must then include this secret value as protected/YOURSECRET/ after the ns/YOURNAMESPACE/ portion. For example, if your webhooks are configured to use https://cloudbees-core.example.com/hibernation/ns/extra-namespace/queue/your-controller/github-webhook/, then the protected form would be https://cloudbees-core.example.com/hibernation/ns/extra-namespace/protected/{secretvalue}/queue/your-controller/github-webhook/ and any accesses to the unprotected form will be rejected with a 401 error without any side effects.

As a convenience, a sidebar link is displayed in any managed controller with hibernation enabled that points to its redirect URL. When protection is enabled, this link automatically includes the secret value. For example, this URL is appropriate to include in a browser bookmark list.

Install the CloudBees managed controller Hibernation Plugin on a managed controller

There are two different ways to install the CloudBees managed controller Hibernation Plugin on a managed controller:

When setting up the managed controller, from the managed controller’s setup wizard.
From the Available tab in Manage Jenkins Manage Plugins.

To install the CloudBees managed controller Hibernation Plugin from a managed controller’s setup wizard:

On the Customize CloudBees CI managed controller screen, select Select Plugins to install.
Locate the CloudBees managed controller Hibernation Plugin entry, and then select the checkbox next to it to select the plugin for installation. Default plugins are shown in green; you should generally leave these selected. These plugins are installed at the same time as the CloudBees managed controller Hibernation Plugin.
Select Install to install the selected plugins.
After plugins have been installed, select the Installed tab and confirm that the CloudBees managed controller Hibernation Plugin is both available and selected.

To install the CloudBees managed controller Hibernation Plugin on an existing managed controller:

On the operations center dashboard, select the name of the managed controller.
Select Manage Jenkins Manage Plugins.
On the Available tab, locate the CloudBees managed controller Hibernation Plugin entry, and then select the checkbox next to it to select the plugin for installation.
Select the appropriate installation option (install without restart or download and install after restart).
After the plugin is installed, select the Installed tab and make sure that the plugin is both available and selected.

Enable hibernation on a managed controller

To enable the hibernation feature on each managed controller:

From the Operations Dashboard, select the name of a managed controller.
Select Manage Jenkins Configure System.
Scroll down to the Automatic Hibernation section.
Select Automatic Hibernation.
Select Save to save the configuration.

Configure hibernation

After the hibernation monitoring service has been enabled in the Helm release and the hibernation plugin has been installed on at least one managed controller, hibernation may be configured on each managed controller to suit your purposes.

To configure hibernation:

Select Manage Jenkins Configure System.
Scroll down to the Automatic Hibernation section.
Complete the following options:
- Enable hibernation?: This option enables or disables automatic hibernation for the managed controller.
- Grace period (seconds): This option configures the number of seconds the instance should wait after apparent activity has ceased before hibernating a given managed controller.
- Running builds: A selector that instructs the instance to treat a managed controller as busy when it is running a build. This option may be disabled by selecting Delete. If this option is disabled, it may be restored by selecting Add, and then selecting Running builds.
- Web browsing: A selector that instructs the instance to treat a managed controller as busy if there is web browser activity against the instance’s web UI. This option may be disabled by selecting Delete. If this option is disabled, it may be restored by selecting Add, and selecting Web browsing.
- Advanced: A sub-option for Web browsing that allows you to specify a Java-format regular expression of URIs to ignore for purposes of activity.
- Add: Restores the Running builds and/or Web browsing options if those options have been deleted.

Track hibernation status

The hibernation status of a given managed controller can be viewed in the operations center dashboard:

Hibernated managed controllers display with a "pause" icon and the word "Hibernated" when hovered over with the mouse pointer.
Selecting the managed controller starts it.

Use hibernation API endpoints

The CloudBees managed controller Hibernation Plugin features three API endpoints that can be used to interact with a hibernated managed controller.

POST queue for hibernation

The POST queue endpoint sends data to a managed controller with hibernation enabled:

If the managed controller is running, the request is proxied (including the path, headers, and request body) and synchronously forwarded.
If the managed controller is hibernating, the managed controller is started in the background, and the request is responded to with a blank 202 Accepted message. Once the managed controller is running, the request is proxied and any response discarded.

For example, to invoke a parameterized build:

$ curl -u <username>:<API token> -d <parameters>=<values> {domain-name}/hibernation/queue/your-controller/job/your-job/buildWithParameters▼

The -d option implies use of the POST method.

Or to run a Groovy script:

$ curl -u <username>:<API token> -d 'script=Jenkins.instance.systemMessage="awoken"' {domain-name}/hibernation/queue/your-controller/script▼

POST queue may be used with GitHub webhooks: refer to Using POST queue with GitHub webhooks.

GET proxy for hibernation

The GET proxy endpoint requests data from a managed controller with hibernation enabled:

If the managed controller is running, the request is proxied (including the path, headers and request body) and synchronously forwarded.
If the managed controller is hibernating, the controller is started in the background, and the request is proxied and responded to.

The managed controller enforces authentication, and some operations may require an authenticated request.

$ curl -u <username>:<API token> -s {domain-name}/hibernation/proxy/your-controller/job/your-job/lastStableBuild/api/json\?tree=number▼

The GET proxy endpoint can be useful for read-only APIs. For example, to download a set of build artifacts from a job on a managed controller, you can use https://cloudbees-core.example.com/hibernation/proxy/your-team/job/your-project/lastStableBuild/artifact/zip/artifacts.zip instead of having to pass a fully authenticated request with a user + API token to https://cloudbees-core.example.com/your-team/job/your-project/lastStableBuild/artifact/zip/artifacts.zip

POST requests for mutating APIs are not supported by this endpoint; such requests are currently supported only by the queue endpoint.

GET redirect for hibernation

The GET redirect endpoint, when issued against a managed controller with hibernation enabled, generates an HTML page that redirects to the managed controller:

If the managed controller is running, the redirect is almost immediate.
If the managed controller is hibernating, the managed controller is started in the background, and once the managed controller is running, the page redirects the request.

The GET redirect endpoint is useful for redirecting browser bookmarks. For example, instead of bookmarking https://cloudbees-core.example.com/your-team/job/your-folder/, you could use the more useful https://cloudbees-core.example.com/hibernation/redirect/your-team/job/your-folder/.

Use POST queue with GitHub webhooks

The POST queue endpoint may be used as part of the URL for GitHub webhooks. When a GitHub webhook arrives at a hibernated managed controller, the managed controller is started, and, once it is fully operational, the webhook is processed. By default, managed controllers are woken by anything that comes into the /queue/ URL except non-matching GitHub events. "Matching" means any known event type (like a new PR) against organizations or repositories defined by the GitHub branch source plugin.

For example, when a PR is filed that corresponds to a project defined in a hibernated managed controller, the webhook "wakes up" the managed controller. After approximately 90 seconds, when the managed controller is ready, it executes and runs the build, and after a few more minutes returns to the hibernated state.

Hyperlinks from GitHub commit statuses using the continuous-integration/jenkins prefix may not work with a hibernated managed controller. However, if you receive a 503 error from a GitHub commit status hyperlink, you can add the redirect infix to the URL to get an actual response. Refer to GET redirect for more information. Hyperlinks produced by the CloudBees GitHub Reporting plugin will automatically include a redirect infix when applicable, so these can be followed whether the managed controller is hibernated or not.

To use the POST queue API endpoint in a GitHub webhook, specify the API endpoint as part of the URL. For example, a URL reading https://cloudbees-core.example.com/your-team/github-webhook/ can be specified as https://cloudbees-core.example.com/hibernation/queue/your-team/github-webhook/.

When CloudBees CI is creating a GitHub webhook on your behalf, or when you use the CloudBees CI wizard to create a GitHub app (which includes webhook functionality), a queue infix will be included in the webhook URL automatically when hibernation is enabled for that managed controller.

Health checks

When controllers are hibernated, external health check systems that use the controller URL can be confused by this. The hibernation monitor offers a controller health check endpoint that takes into account the hibernation lifecycle and returns HTTP response codes accordingly.

The health check endpoint is exposed at https://cloudbees-core.example.com/hibernation/ns/your-namespace/health/your-controller/, and possible responses are:

Table 1. Hibernation health check endpoint responses
Response	Description
`HTTP 200 OK { "state": "READY" }` ▼	The controller is up and running (not hibernated).
`HTTP 200 OK { "state": "HIBERNATED" }` ▼	The controller is hibernated.
`HTTP 200 OK { "state": "HIBERNATING" }` ▼	The controller is in the process of hibernating.
`HTTP 200 OK { "state": "WAKING_UP" }` ▼	The controller is waking up from hibernation.
`HTTP 503 SERVICE_UNAVAILABLE { "state": "WAKING_UP", "failure": "Controller not waking up in time (N seconds)" }` ▼	The controller is taking too long to wake up.
`HTTP 500 INTERNAL_SERVER_ERROR { "state": "UNKNOWN", "failure": "Cannot compute controller state" }` ▼	There is a problem retrieving the state.

Wake-up time is considered too long when it takes more than the established Kubernetes readiness probe failure threshold for the controller. This is, by default, 530 seconds.

Controller Lifecycle Notifications

You can use the Controller Lifecycle Notifications plugin to send webhooks from the operations center to a configured endpoint based on the managed controller state changes, such as provisioned and deprovisioned.

For information about the Controller Lifecycle Notifications plugin, refer to Controller Lifecycle Notifications Plugin.