Managing controllers

8 minute read

When new teams join an organization, or existing teams start a new project, CloudBees CI makes it easy to provision a fully managed and access controlled controller per team. In CloudBees CI, a controller is referred to as a managed controller.

Administrators can provision managed controllers from a standardized template or they can allow team leads to provision their own managed controllers "on-demand." The number of controllers an environment can handle is limited only by the capacity of the cluster.

Upgrading managed controllers

To update a managed controller’s version, the administrator needs to update the version of the Docker image for the managed controller. Once this version is updated, the managed controller and its default plugins will be upgraded to the latest versions defined in the image, while any non-default plugins will be left at their current versions.

Once the Docker image definition is updated, an administrator needs to restart the instance so the managed controller can begin using its upgraded components.

Bulk upgrades

When a cluster serves many teams and contains many controllers, an administrator can save time and greatly reduce the overhead of upgrading those controllers by creating a repeatable task to automate this process. This can be achieved by defining a cluster operation in operations center.

To create a task to perform a bulk upgrade:

  1. Sign in to operations center as an administrator.

  2. Create a New Item for Cluster Operations.

  3. Select the managed controllers cluster operation

  4. Select the pre-configured upgrade pattern that you want to use.

  5. In the Uses Docker Image, pick a Docker image that is used by the cluster’s controllers as the upgrade target. Any controller in the cluster using the selected image will be affected by this cluster operation.

  6. In the Steps section, select Backup Controller and configure the options as per Using the CloudBees Backup plugin

  7. In the Steps section, select Update Docker Image and then pick the new Docker to which you want to bulk-upgrade the targeted controllers.

  8. Add a Reprovision step to restart the targeted controllers.

Once these settings are configured, the administrator can run the cluster operation to perform a bulk upgrade of their cluster’s controllers, or schedule the operation to run at a later time.

Optimizing resources with hibernation of managed controllers

A common issue in installations with many managed controllers is having a cluster with a lot of managed controllers that aren’t being used most of the time. What typically happens in large organizations is multiple controllers are heavily used during working hours, and then sit idle during non-working hours. The idle controllers still cost the same even though they’re not being used. Similarly static agents where the work has been completed can sit entirely unused and still occupy valuable infrastructure resources. Without the ability to load-balance overtaxed controllers and remove unused static agents, these resources can passively increase cloud spending and infrastructure resources. Large-scale organizations can end up paying significant amounts of money for idle and unused resources.

For those cases, CloudBees offers the managed controller hibernation feature, which helps you "turn off" idle or unused managed controllers.

For example, if you have managed controllers that are only being used by your development teams during business hours, you can reclaim those resources when they’re idle instead of having those managed controllers "up" around the clock.

Installing and enabling hibernation

To use hibernation, you need to take three actions:

  1. Enable hibernation in the Helm chart.

  2. Install the CloudBees managed controller Hibernation Plugin on any managed controller you want to be able to hibernate.

  3. Enable the CloudBees managed controller Hibernation Plugin after installation.

Enabling the hibernation monitor using Helm

Use the helm upgrade command to enable the managed controller Hibernation monitoring service in an existing Helm release of CloudBees CI.

  1. If you have not already done so, create a custom values YAML file to store customizations for your instance. For instructions, see Using a custom YAML file to install CloudBees CI.

  2. Add the following clause to your custom values file:

    Hibernation: Enabled: true
  3. Run Helm upgrade with the following command:

    $ helm upgrade -f <name-of-custom-values-file>.yaml cloudbees-core <path-to-CB-helm-chart>

Enabling hibernation in a separate namespace

If you are following the instructions for adding a specific Namespace with Helm section of “Managed controllers in specific Kubernetes namespaces” and you wish to enable hibernation for at least some controllers in that namespace, add the following to the values.yaml in those instructions.

OperationsCenter: Enabled: false HostName: cloudbees-core.example.com (1) Ingress: Class: nginx (2) Hibernation: Enabled: true
1 Replace cloudbees-core.example.com with your domain name.
2 Replace nginx with your Ingress class name

A separate instance of the monitoring service is created in that namespace and handles controllers only in the same namespace. An ingress is also created in that namespace to make namespaced hibernation links accessible.

Since the generated ingresses will share a single host name, all API endpoints mentioned below support an infix ns/extra-namespace/ after hibernation/, which routes requests to the monitor running in a specific namespace.

Where appropriate, this infix is included automatically in URLs supplied by CloudBees CI:

  • A hibernated managed controller’s link in the operations center dashboard may look like: https://cloudbees-core.example.com/hibernation/ns/extra-namespace/redirect/your-master/

  • The hook URL registered when creating a GitHub organization folder may look like: https://cloudbees-core.example.com/hibernation/ns/extra-namespace/queue/your-master/github-webhook/

Installing the CloudBees managed controller Hibernation Plugin on a managed controller

There are two different ways to install the CloudBees managed controller Hibernation Plugin on a managed controller:

  • When setting up the managed controller, from the managed controller’s setup wizard.

  • From the Available tab in Manage Jenkins  Manage Plugins.

To install the CloudBees managed controller Hibernation Plugin from a managed controller’s setup wizard:

  1. On the Customize CloudBees CI managed controller screen, click Select Plugins to install.

  2. Locate the CloudBees managed controller Hibernation Plugin entry, and then select the checkbox next to it to select the plugin for installation. Default plugins are shown in green; you should generally leave these selected. These plugins are installed at the same time as the CloudBees managed controller Hibernation Plugin.

  3. Click Install to install the selected plugins.

  4. After plugins have been installed, check the Installed tab and confirm that the CloudBees managed controller Hibernation Plugin is both available and selected.

To install the CloudBees managed controller Hibernation Plugin on an existing managed controller:

  1. On the operations center dashboard, select the name of the managed controller.

  2. Click Manage Jenkins  Manage Plugins.

  3. On the Available tab, locate the CloudBees managed controller Hibernation Plugin entry, and then select the checkbox next to it to select the plugin for installation.

  4. Click the appropriate installation option (install without restart or download and install after restart).

  5. After the plugin is installed, check the Installed tab and make sure that the plugin is both available and selected.

Enabling hibernation on a managed controller

To enable the hibernation feature on each managed controller:

  1. From the Operations Dashboard, select the name of a managed controller.

  2. Click Manage Jenkins  Configure System.

  3. Scroll down to the Automatic Hibernation section.

  4. Click to select the Automatic Hibernation feature.

  5. Click Save to save the configuration.

Configuring hibernation

After the hibernation monitoring service has been enabled in the Helm release and the hibernation plugin has been installed on at least one managed controller, hibernation may be configured on each managed controller to suit your purposes.

To configure hibernation:

  1. Select Manage Jenkins  Configure System.

  2. Scroll down to the Automatic Hibernation section.

  3. Complete the following options:

    • Enable hibernation?: This option enables or disables automatic hibernation for the managed controller.

    • Grace period (seconds): This option configures the number of seconds the instance should wait after apparent activity has ceased before hibernating a given managed controller.

    • Running builds: A selector that instructs the instance to treat a managed controller as busy when it is running a build. This option may be disabled by clicking Delete. If this option is disabled, it may be restored by clicking Add, and then selecting Running builds.

    • Web browsing: A selector that instructs the instance to treat a managed controller as busy if there is web browser activity against the instance’s web UI. This option may be disabled by clicking Delete. If this option is disabled, it may be restored by clicking Add, and selecting Web browsing.

    • Advanced: A sub-option for Web browsing that allows you to specify a Java-format regular expression of URIs to ignore for purposes of activity.

    • Add: Restores the Running builds and/or Web browsing options if those options have been deleted.

Tracking hibernation status

The hibernation status of a given managed controller can be seen in the operations center dashboard:

  • Hibernated managed controllers display with a blue "pause" icon:

    hibernated master pause

    And the word "Hibernated" when hovered over with the mouse pointer.

  • Clicking on the managed controller starts it.

Using hibernation API endpoints

The CloudBees managed controller Hibernation Plugin features three API endpoints that can be used to interact with a hibernated managed controller.

POST queue for hibernation

The POST queue endpoint sends data to a managed controller with hibernation enabled:

  • If the managed controller is running, the request is proxied (including the path, headers, and request body) and synchronously forwarded.

  • If the managed controller is hibernating, the managed controller is started in the background, and the request is responded to with a blank 202 Accepted message. Once the managed controller is running, the request is proxied and any response discarded.

For example, to invoke a parameterized build:

$ curl -u <username>:<API token> -d <parameters>=<values> {domain-name}/hibernation/queue/your-master/job/your-job/buildWithParameters
The -d option implies use of the POST method.

Or to run a Groovy script:

$ curl -u <username>:<API token> -d 'script=Jenkins.instance.systemMessage="awoken"' {domain-name}/hibernation/queue/your-master/script

POST queue may be used with GitHub webhooks: see Using POST queue with GitHub webhooks.

GET proxy for hibernation

The GET proxy endpoint requests data from a managed controller with hibernation enabled:

  • If the managed controller is running, the request is proxied (including the path, headers and request body) and synchronously forwarded.

  • If the managed controller is hibernating, the controller is started in the background, and the request is proxied and responded to.

The managed controller enforces authentication, and some operations may require an authenticated request.
$ curl -u <username>:<API token> -s {domain-name}/hibernation/proxy/your-master/job/your-job/lastStableBuild/api/json\?tree=number

The GET proxy endpoint can be useful for read-only APIs. For example, to download a set of build artifacts from a job on a managed controller, you can use https://cloudbees-core.example.com/hibernation/proxy/your-team/job/your-project/lastStableBuild/artifact/zip/artifacts.zip instead of having to pass a fully authenticated request with a user + API token to https://cloudbees-core.example.com/your-team/job/your-project/lastStableBuild/artifact/zip/artifacts.zip

POST requests for mutating APIs are not supported by this endpoint; such requests are currently supported only by the queue endpoint.

GET redirect for hibernation

The GET redirect endpoint, when issued against a managed controller with hibernation enabled, generates an HTML page that redirects to the managed controller:

  • If the managed controller is running, the redirect is almost immediate.

  • If the managed controller is hibernating, the managed controller is started in the background, and once the managed controller is running, the page redirects the request.

The GET redirect endpoint is useful for redirecting browser bookmarks. For example, instead of bookmarking https://cloudbees-core.example.com/your-team/job/your-folder/, you could use the more useful https://cloudbees-core.example.com/hibernation/redirect/your-team/job/your-folder/.

Using POST queue with GitHub webhooks

The POST queue endpoint may be used as part of the URL for GitHub webhooks. When a GitHub webhook arrives at a hibernated managed controller, the managed controller is started, and, once it is fully operational, the webhook is processed. By default, managed controllers are woken by anything that comes into the /queue/ URL except non-matching GitHub events. "Matching" means any known event type (like a new PR) against organizations or repositories defined by the GitHub branch source plugin.

For example, when a PR is filed that corresponds to a project defined in a hibernated managed controller, the webhook "wakes up" the managed controller. After approximately 90 seconds, when the managed controller is ready, it executes and runs the build, and after a few more minutes returns to the hibernated state.

Hyperlinks from GitHub commit statuses using the continuous-integration/jenkins prefix may not work with a hibernated managed controller. However, if you receive a 503 error from a GitHub commit status hyperlink, you can add the redirect infix to the URL to get an actual response. See GET redirect for more information. Hyperlinks produced by the CloudBees GitHub Reporting plugin will automatically incude a redirect infix when applicable, so these can be followed whether the managed controller is hibernated or not.

To use the POST queue API endpoint in a GitHub webhook, specify the API endpoint as part of the URL. For example, a URL reading https://cloudbees-core.example.com/your-team/github-webhook/ can be specified as https://cloudbees-core.example.com/hibernation/queue/your-team/github-webhook/.

When CloudBees CI is creating a GitHub webhook on your behalf, or when you use the CloudBees CI wizard to create a GitHub app (which includes webhook functionality), a queue infix will be included in the webhook URL automatically when hibernation is enabled for that managed controller.
In August 2020, the Jenkins project voted to replace the term master with controller. We have taken a pragmatic approach to cleaning these up, ensuring the least amount of downstream impact as possible. CloudBees is committed to ensuring a culture and environment of inclusiveness and acceptance - this includes ensuring the changes are not just cosmetic ones, but pervasive. As this change happens, please note that the term master has been replaced through the latest versions of the CloudBees documentation with controller (as in managed controller, client controller, team controller) except when still used in the UI or in code.