Managing Masters

When new teams join an organization, or existing teams start a new project, CloudBees Core makes it easy to provision a fully managed and access controlled Jenkins master per team. In CloudBees Core, a Jenkins master is referred to as a Managed Master.

Administrators can provision Managed Masters from a standardized template or they can allow team leads to provision their own Managed Masters "on-demand." The number of masters a CloudBees Core environment can handle is limited only by the capacity of the cluster.

Upgrading Managed Masters

To update a Managed Master’s version, the administrator needs to update the version of the Docker image for the Managed Master. Once this version is updated, the Managed Master and its default plugins will be upgraded to the latest versions defined in the image, while any non-default plugins will be left at their current versions.

master configuration overview 2

Once the Docker image definition is updated, the CloudBees Core administrator will need to restart the instance so the Managed Master can begin using its upgraded components.

action stop
action stop confirm
action start

Bulk upgrades

When a CloudBees Core cluster serves many teams and contains many masters, an administrator can save time and greatly reduce the overhead of upgrading those masters by creating a repeatable task to automate this process. In CloudBees Core, this can be achieved by defining a cluster operation in Operations Center.

To create a task to perform a bulk upgrade:

  1. Log into Operations Center as an administrator.

  2. Create a New Item for Cluster Operations.

  3. Select the Managed Masters cluster operation

  4. Select the pre-configured upgrade pattern that you want to use.

  5. In the Uses Docker Image, pick a Docker image that is used by the cluster’s masters as the upgrade target. Any master in the cluster using the selected image will be affected by this cluster operation.

  6. In the Steps section, select Update Docker Image and then pick the new Docker to which you want to bulk-upgrade the targeted masters.

  7. Add a Reprovision step to restart the targeted masters.

Once these settings are configured, the administrator can run the cluster operation to perform a bulk upgrade of their cluster’s masters, or schedule the operation to run at a later time.

Hibernation in managed masters

A common issue in installations with many masters is "I have a cluster with a lot of masters that aren’t being used most of the time". For those cases, CloudBees offers the managed master hibernation feature, which helps you "turn off" idle or unused masters.

For example, if you have masters that are only being used by your development teams during business hours, you can reclaim those resources when they’re idle instead of having those masters "up" around the clock.

Managed master hibernation is available only in CloudBees Core on modern cloud platforms, and only as a technical preview.

Hibernation support status

As of October 2019, managed master hibernation is a technical preview.

A Technical Preview feature:

  • Has not undergone end-to-end testing with CloudBees products.

  • Is not bundled with the product, but can be installed from the CloudBees update center.

  • Is supported by CloudBees for issues related to the feature, subject to CloudBees internal priorities.

  • Is supported by CloudBees for compatibility issues, subject to CloudBees internal priorities.

Installing and enabling hibernation

To use the technical preview version of hibernation, you need to take three actions:

  1. Enable hibernation in the Helm chart.

  2. Install the CloudBees Managed Master Hibernation Plugin on any master you want to be able to hibernate.

  3. Enable the CloudBees Managed Master Hibernation Plugin after installation.

Enabling the hibernation monitor using Helm

Use the helm upgrade command to enable the Managed Master Hibernation monitoring service in an existing Helm release of CloudBees Core.

  1. If you have not already done so, create a custom values YAML file to store customizations for your instance. For instructions, see the GitOps and Custom Values Files section of "Installing Kubernetes using Helm".

  2. Add the following clause to your custom values file:

    Hibernation:
      Enabled: true
  3. Upgrade Helm with the following command:

    $ helm upgrade -f <name-of-custom-values-file>.yaml cloudbees-core <path-to-CB-helm-chart>

Installing the CloudBees Managed Master Hibernation Plugin on a master

There are two different ways to install the CloudBees Managed Master Hibernation Plugin on a managed master:

  • When setting up the master, from the master’s setup wizard.

  • From the Available tab in Manage Jenkins  Manage Plugins.

To install the CloudBees Managed Master Hibernation Plugin from a managed master’s setup wizard:

  1. On the "Customize CloudBees Core Managed Master" screen, click on Select Plugins to install.

  2. Locate the CloudBees Managed Master Hibernation Plugin entry and click the checkbox to select the plugin for installation.

  3. Default plugins are shown in green; you should generally leave these selected. These plugins will be installed at the same time as the CloudBees Managed Master Hibernation Plugin.

  4. Click Install to install the selected plugins.

  5. After plugins have been installed, check the Installed tab and confirm that the CloudBees Managed Master Hibernation Plugin is both available and selected.

To install the CloudBees Managed Master Hibernation Plugin on an existing managed master:

  1. On the Operations Center dashboard, click on the name of the managed master.

  2. Click Manage Jenkins  Manage Plugins.

  3. On the Available tab, locate the CloudBees Managed Master Hibernation Plugin entry and click the checkbox to select the plugin for installation.

  4. Click the appropriate installation option (install without restart or download and install after restart).

  5. After the plugin has been installed, check the Installed tab and make sure that the plugin is both available and selected.

Enabling hibernation on a managed master

To enable the hibernation feature on each managed master:

  1. From the Operations Dashboard, click on the name of a managed master.

  2. Click Manage Jenkins  Configure System.

  3. Scroll down to the Automatic Hibernation section.

  4. Click to select the "Automatic Hibernation" feature.

  5. Click on the Save button to save the configuration.

Configuring hibernation

After the hibernation monitoring service has been enabled in the Helm release and the hibernation plugin has been installed on at least one managed master, hibernation may be configured on each managed master to suit your purposes.

To configure hibernation, click on Manage Jenkins  Configure System. Scroll down to the Automatic Hibernation section.

The Automatic Hibernation feature has six selectable options:

mm hibernation main annotated
  1. Enable hibernation?: This option enables or disables automatic hibernation for the master.

  2. Grace period (seconds): This option configures the number of seconds the instance should wait after apparent activity has ceased before hibernating a given master.

  3. Running builds: A selector that instructs the instance to treat a master as busy when it is running a build. This option may be disabled by clicking on Delete. If this option is disabled, it may be restored by clicking on Add (6) and selecting Running builds.

  4. Web browsing: A selector that instructs the instance to treat a master as busy if there is web browser activity against the instance’s web UI. This option may be disabled by clicking on Delete. If this option is disabled, it may be restored by clicking on Add (6) and selecting Web browsing.

  5. Advanced: A sub-option for Web browsing that allows you to specify a Java-format regular expression of URIs to ignore for purposes of activity.

  6. Add: Restores the Running builds and/or Web browsing options if those options have been deleted.

Tracking hibernation status

The hibernation status of a given master can be seen in the Operations Center dashboard:

  • Hibernated masters will display with a blue "pause" icon:

    hibernated master pause

    And the word "Hibernated" when hovered over with the mouse pointer.

  • Clicking on the master starts the master.

Using hibernation API endpoints

The CloudBees Managed Master Hibernation Plugin features three API endpoints that can be used to interact with a hibernated master.

POST queue

The POST queue endpoint sends data to a managed master with hibernation enabled:

  • If the managed master is running, the request is proxied (including the path, headers, and request body) and synchronously forwarded.

  • If the managed master is hibernating, the master is started in the background, and the request is responded to with a blank 202 Accepted message. Once the managed master is running, the request is proxied and any response discarded.

For example, to invoke a parameterized build:

$ curl -u <username>:<API token> -d <parameters>=<values> <URL of instance>/hibernation/queue/your-master/job/your-job/buildWithParameters

POST queue may be used with GitHub webhooks: see Using POST queue with GitHub webhooks.

GET proxy

The GET proxy endpoint requests data from a managed master with hibernation enabled:

  • If the managed master is running, the request is proxied (including the path, headers and request body) and synchronously forwarded.

  • If the managed master is hibernating, the master is started in the background, and the request is proxied and responded to.

POST requests for mutating APIs are not supported.

The managed master enforces authentication, and some operations may require an authenticated request.
$ curl -u <username>:<API token> -s <URL of instance>/hibernation/proxy/your-master/job/your-job/lastStableBuild/api/json\?tree=number

The GET proxy endpoint can be useful for read-only APIs. For example, to download a set of build artifacts from a job on a managed master, you can use https://cloudbees-core-instance.yourcompany.com/hibernation/proxy/your-team/job/your-project/lastStableBuild/artifact/zip/artifacts.zip instead of having to pass a fully authenticated request with a user + API token to https://cloudbees-core-instance.yourcompany.com/your-team/job/your-project/lastStableBuild/artifact/zip/artifacts.zip

GET redirect

The GET redirect endpoint, when issued against a managed master with hibernation enabled, generates an HTML page that redirects to the managed master:

  • If the managed master is running, the redirect is almost immediate.

  • If the managed master is hibernating, the master is started in the background, and once the managed master is running, the page redirects the request.

The GET redirect endpoint is useful for redirecting browser bookmarks. For example, instead of bookmarking https://<URL of instance>/your-team/job/your-folder/, you could use the more useful https://<URL of instance>/hibernation/redirect/your-team/job/your-folder/.

Using POST queue with GitHub webhooks

The POST queue endpoint may be used as part of the URL for GitHub webhooks. When a GitHub webhook arrives at a hibernated master, the master is started, and, once it is fully operational, the webhook is processed. By default, masters are woken by anything that comes into the /queue/ URL except non-matching GitHub events. "Matching" means any known even type (like a new PR) against organizations or repositories defined by the GitHub branch source plugin.

For example, when a PR is filed that corresponds to a project defined in a hibernated master, the webhook "wakes up" the master. After approximately 90 seconds, when the master is ready, it executes and runs the build, and after a few more minutes returns to the hibernated state.

In the technical preview, GitHub status hyperlinks may not work with a hibernated master. However, if you receive a 503 error from a GitHub status hyperlink, you can add the redirect parameter to the URL to get an actual response. See GET redirect for more information.

To use the POST queue API endpoint in a GitHub webhook, specify the API endpoint as part of the URL. For example, a URL reading https://cloudbees-core-instance.yourcompany.com/your-team/github-webhook can be specified as https://cloudbees-core-instance.yourcompany.com/hibernation/queue/your-team/github-webhook.

Copyright © 2010-2019 CloudBees, Inc.Online version published by CloudBees, Inc. under the Creative Commons Attribution-ShareAlike 4.0 license.CloudBees and CloudBees DevOptics are registered trademarks and CloudBees Core, CloudBees CodeShip, CloudBees Jenkins Enterprise, CloudBees Jenkins Platform, CloudBees Jenkins Operations Center and DEV@cloud are trademarks of CloudBees, Inc. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Jenkins is a registered trademark of the non-profit Software in the Public Interest organization. Used with permission. See here for more info about the Jenkins project. The registered trademark Jenkins® is used pursuant to a sublicense from the Jenkins project and Software in the Public Interest, Inc. Read more at www.cloudbees.com/jenkins/about. Apache, Apache Ant, Apache Maven, Ant and Maven are trademarks of The Apache Software Foundation. Used with permission. No endorsement by The Apache Software Foundation is implied by the use of these marks.Other names may be trademarks of their respective owners. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this content, and CloudBees was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this content, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.