Managing Masters

When new teams join an organization, or existing teams start a new project, CloudBees Core makes it easy to provision a fully managed and access controlled Jenkins master per team. In CloudBees Core, a Jenkins master is referred to as a Managed Master.

Administrators can provision Managed Masters from a standardized template or they can allow team leads to provision their own Managed Masters "on-demand." The number of masters a CloudBees Core environment can handle is limited only by the capacity of the cluster.

Upgrading Managed Masters

To update a Managed Master’s version, the administrator needs to update the version of the Docker image for the Managed Master. Once this version is updated, the Managed Master and its default plugins will be upgraded to the latest versions defined in the image, while any non-default plugins will be left at their current versions.

master configuration overview 2

Once the Docker image definition is updated, the CloudBees Core administrator will need to restart the instance so the Managed Master can begin using its upgraded components.

action stop
action stop confirm
action start

Bulk upgrades

When a CloudBees Core cluster serves many teams and contains many masters, an administrator can save time and greatly reduce the overhead of upgrading those masters by creating a repeatable task to automate this process. In CloudBees Core, this can be achieved by defining a cluster operation in Operations Center.

To create a task to perform a bulk upgrade:

  1. Log into Operations Center as an administrator.

  2. Create a New Item for Cluster Operations.

  3. Select the Managed Masters cluster operation

  4. Select the pre-configured upgrade pattern that you want to use.

  5. In the Uses Docker Image, pick a Docker image that is used by the cluster’s masters as the upgrade target. Any master in the cluster using the selected image will be affected by this cluster operation.

  6. In the Steps section, select Backup Master and configure the options as per Using the CloudBees Backup plugin

  7. In the Steps section, select Update Docker Image and then pick the new Docker to which you want to bulk-upgrade the targeted masters.

  8. Add a Reprovision step to restart the targeted masters.

Once these settings are configured, the administrator can run the cluster operation to perform a bulk upgrade of their cluster’s masters, or schedule the operation to run at a later time.

Hibernation in Managed Masters

A common issue in installations with many Managed Masters is having a cluster with a lot of Managed Masters that aren’t being used most of the time. For those cases, CloudBees offers the Managed Master hibernation feature, which helps you "turn off" idle or unused Managed Masters.

For example, if you have Managed Masters that are only being used by your development teams during business hours, you can reclaim those resources when they’re idle instead of having those Managed Masters "up" around the clock.

Installing and enabling hibernation

To use hibernation, you need to take three actions:

  1. Enable hibernation in the Helm chart.

  2. Install the CloudBees Managed Master Hibernation Plugin on any Managed Master you want to be able to hibernate.

  3. Enable the CloudBees Managed Master Hibernation Plugin after installation.

Enabling the hibernation monitor using Helm

Use the helm upgrade command to enable the Managed Master Hibernation monitoring service in an existing Helm release of CloudBees Core.

  1. If you have not already done so, create a custom values YAML file to store customizations for your instance. For instructions, see Using a custom YAML file to install CloudBees Core.

  2. Add the following clause to your custom values file:

    Hibernation:
      Enabled: true
  3. Upgrade Helm with the following command:

    $ helm upgrade -f <name-of-custom-values-file>.yaml cloudbees-core <path-to-CB-helm-chart>

Enabling hibernation in a separate namespace

If you are following the instructions for adding a specific Namespace with Helm section of “Managed Masters in specific Kubernetes namespaces” and you wish to enable hibernation for at least some masters in that namespace, add the following to the values.yaml in those instructions.

Hibernation:
  Enabled: true

A separate instance of the monitoring service is created in that namespace and handles masters only in the same namespace.

Since the generated ingresses will share a single host name, all API endpoints mentioned below support an infix ns/<name of namespace>/ after hibernation/, which routes requests to the monitor running in a specific namespace.

Where appropriate, this infix is included automatically in URLs supplied by CloudBees Core:

  • A hibernated Managed Master’s link in the Operations Center dashboard may look like: https://cloudbees-core-instance.yourcompany.com/hibernation/ns/extra-namespace/redirect/your-master/

  • The hook URL registered when creating a GitHub organization folder may look like: https://cloudbees-core-instance.yourcompany.com/hibernation/ns/extra-namespace/queue/your-master/github-webhook/

Installing the CloudBees Managed Master Hibernation Plugin on a Managed Master

There are two different ways to install the CloudBees Managed Master Hibernation Plugin on a Managed Master:

  • When setting up the Managed Master, from the Managed Master’s setup wizard.

  • From the Available tab in Manage Jenkins  Manage Plugins.

To install the CloudBees Managed Master Hibernation Plugin from a Managed Master’s setup wizard:

  1. On the Customize CloudBees Core Managed Master screen, click Select Plugins to install.

  2. Locate the CloudBees Managed Master Hibernation Plugin entry, and then select the checkbox next to it to select the plugin for installation. Default plugins are shown in green; you should generally leave these selected. These plugins are installed at the same time as the CloudBees Managed Master Hibernation Plugin.

  3. Click Install to install the selected plugins.

  4. After plugins have been installed, check the Installed tab and confirm that the CloudBees Managed Master Hibernation Plugin is both available and selected.

To install the CloudBees Managed Master Hibernation Plugin on an existing Managed Master:

  1. On the Operations Center dashboard, select the name of the Managed Master.

  2. Click Manage Jenkins  Manage Plugins.

  3. On the Available tab, locate the CloudBees Managed Master Hibernation Plugin entry, and then select the checkbox next to it to select the plugin for installation.

  4. Click the appropriate installation option (install without restart or download and install after restart).

  5. After the plugin is installed, check the Installed tab and make sure that the plugin is both available and selected.

Enabling hibernation on a Managed Master

To enable the hibernation feature on each Managed Master:

  1. From the Operations Dashboard, select the name of a Managed Master.

  2. Click Manage Jenkins  Configure System.

  3. Scroll down to the Automatic Hibernation section.

  4. Click to select the Automatic Hibernation feature.

  5. Click Save to save the configuration.

Configuring hibernation

After the hibernation monitoring service has been enabled in the Helm release and the hibernation plugin has been installed on at least one Managed Master, hibernation may be configured on each Managed Master to suit your purposes.

To configure hibernation:

  1. Select Manage Jenkins  Configure System.

  2. Scroll down to the Automatic Hibernation section.

  3. Complete the following options:

    • Enable hibernation?: This option enables or disables automatic hibernation for the Managed Master.

    • Grace period (seconds): This option configures the number of seconds the instance should wait after apparent activity has ceased before hibernating a given Managed Master.

    • Running builds: A selector that instructs the instance to treat a Managed Master as busy when it is running a build. This option may be disabled by clicking Delete. If this option is disabled, it may be restored by clicking Add, and then selecting Running builds.

    • Web browsing: A selector that instructs the instance to treat a Managed Master as busy if there is web browser activity against the instance’s web UI. This option may be disabled by clicking Delete. If this option is disabled, it may be restored by clicking Add, and selecting Web browsing.

    • Advanced: A sub-option for Web browsing that allows you to specify a Java-format regular expression of URIs to ignore for purposes of activity.

    • Add: Restores the Running builds and/or Web browsing options if those options have been deleted.

Tracking hibernation status

The hibernation status of a given Managed Master can be seen in the Operations Center dashboard:

  • Hibernated Managed Masters display with a blue "pause" icon:

    hibernated master pause

    And the word "Hibernated" when hovered over with the mouse pointer.

  • Clicking on the Managed Master starts it.

Using hibernation API endpoints

The CloudBees Managed Master Hibernation Plugin features three API endpoints that can be used to interact with a hibernated Managed Master.

POST queue for hibernation

The POST queue endpoint sends data to a Managed Master with hibernation enabled:

  • If the Managed Master is running, the request is proxied (including the path, headers, and request body) and synchronously forwarded.

  • If the Managed Master is hibernating, the Managed Master is started in the background, and the request is responded to with a blank 202 Accepted message. Once the Managed Master is running, the request is proxied and any response discarded.

For example, to invoke a parameterized build:

$ curl -u <username>:<API token> -d <parameters>=<values> <URL of instance>/hibernation/queue/your-master/job/your-job/buildWithParameters

POST queue may be used with GitHub webhooks: see Using POST queue with GitHub webhooks.

GET proxy for hibernation

The GET proxy endpoint requests data from a Managed Master with hibernation enabled:

  • If the Managed Master is running, the request is proxied (including the path, headers and request body) and synchronously forwarded.

  • If the Managed Master is hibernating, the master is started in the background, and the request is proxied and responded to.

POST requests for mutating APIs are not supported.

The Managed Master enforces authentication, and some operations may require an authenticated request.
$ curl -u <username>:<API token> -s <URL of instance>/hibernation/proxy/your-master/job/your-job/lastStableBuild/api/json\?tree=number

The GET proxy endpoint can be useful for read-only APIs. For example, to download a set of build artifacts from a job on a Managed Master, you can use https://cloudbees-core-instance.yourcompany.com/hibernation/proxy/your-team/job/your-project/lastStableBuild/artifact/zip/artifacts.zip instead of having to pass a fully authenticated request with a user + API token to https://cloudbees-core-instance.yourcompany.com/your-team/job/your-project/lastStableBuild/artifact/zip/artifacts.zip

GET redirect for hibernation

The GET redirect endpoint, when issued against a Managed Master with hibernation enabled, generates an HTML page that redirects to the Managed Master:

  • If the Managed Master is running, the redirect is almost immediate.

  • If the Managed Master is hibernating, the Managed Master is started in the background, and once the Managed Master is running, the page redirects the request.

The GET redirect endpoint is useful for redirecting browser bookmarks. For example, instead of bookmarking https://<URL of instance>/your-team/job/your-folder/, you could use the more useful https://<URL of instance>/hibernation/redirect/your-team/job/your-folder/.

Using POST queue with GitHub webhooks

The POST queue endpoint may be used as part of the URL for GitHub webhooks. When a GitHub webhook arrives at a hibernated Managed Master, the Managed Master is started, and, once it is fully operational, the webhook is processed. By default, Managed Masters are woken by anything that comes into the /queue/ URL except non-matching GitHub events. "Matching" means any known event type (like a new PR) against organizations or repositories defined by the GitHub branch source plugin.

For example, when a PR is filed that corresponds to a project defined in a hibernated Managed Master, the webhook "wakes up" the Managed Master. After approximately 90 seconds, when the Managed Master is ready, it executes and runs the build, and after a few more minutes returns to the hibernated state.

GitHub status hyperlinks may not work with a hibernated Managed Master. However, if you receive a 503 error from a GitHub status hyperlink, you can add the redirect parameter to the URL to get an actual response. See GET redirect for more information.

To use the POST queue API endpoint in a GitHub webhook, specify the API endpoint as part of the URL. For example, a URL reading https://cloudbees-core-instance.example.com/your-team/github-webhook/ can be specified as https://cloudbees-core-instance.example.com/hibernation/queue/your-team/github-webhook/.