Upgrading CloudBees CI on modern cloud platforms

11 minute read

This guide describes the upgrade process for CloudBees CI on Kubernetes installations. CloudBees recommends that you always upgrade to the latest version of your CloudBees product, ensuring wider lifecycle support coverage, plus more security patches added via the CloudBees Security Advisory, resolved issues, and new features.

Helm is the preferred method of managing CloudBees CI on modern cloud platforms. CloudBees provides a Helm chart for CloudBees CI on modern cloud platforms, and recommends migrating existing installations to use the Helm chart.

  • For controllers that are in your environment:

    • The operations center must be upgraded first, and then controllers can be upgraded.

    • The controller used with the operations center does not have to be at the same version as the operations center. The operations center version must always be more recent or as old as the version of the controllers that are used with the operations center (operations center >= controller version). CloudBees supports upgrading CloudBees CI to a version up to one year later than your current version.

Key principles to follow when upgrading from one version to the other

Change one thing at a time

Limit the changes you carry out to one area at a time. For example, do not plan to make infrastructure changes such as hardware upgrades, networking changes, OS package upgrades at the same time as upgrading Jenkins. This will help in identifying and isolating the root cause of any issues encountered.

Backup strategy

  1. Ensure that you have regular backups of your JENKINS_HOME for your operations center, and all controllers. We recommend using the CloudBees Backup Plugin to automate this. controllers that have CloudBees Backup Plugin 3.38 or later installed have support for the Restore job type. To restore an operations center when using CloudBees CI on traditional platforms follow the steps for Restoring manually. If you are using CloudBees CI on modern cloud platforms, follow the steps to Backup and restore on Kubernetes. The backups should be stored at an offsite location such as cloud storage or at a minimum on separate hardware from your CloudBees CI .

  2. Take another backup immediately before you start the upgrade.

  3. Test that this backup is valid as we use it to recover from a failure if we encounter a severe issue during the production upgrade. A great way to test the validity of these backups is with a test environment. Another reason for testing the builds is that after an upgrade of the Jenkins core version or plugins, you cannot directly downgrade the war file and plugins, the only way to 'roll back' is to restore the JENKINS_HOME backup. To validate the integrity of the backup it is essential to extract the backup to another filesystem, and verify the checksums using:

    find $JENKINS_HOME -type f -print0 | sort -z | xargs -0 md5sum >~/checksum.log.backup

    Run this for both the live and backup data, and compare the checksum.log.* between the two.

Test strategy

  1. We recommend you have a separate test environment with a similar setup to your production environment, with representative jobs. If you are creating a test environment and require a license please send us the instance ID of your test instance and we will send you a test instance license.

  2. We recommend having jobs on your test controller that are representative of jobs on your production controller, so that testing the upgrade, uses the key plugins to your workflow, and the interactions with any external systems (SCM, artifact repositories, bug trackers, cloud automation). It is a best practice to build some jobs which are a good subset of representative jobs on this test cluster. These dedicated jobs represent your production jobs, but without working on your real data. We recommend talking to your development groups to find jobs that cover the main functionality. Here are some recommendations:

    • test your end-to-end complex pipeline jobs

    • test those jobs that utilize critical plugins

    • run jobs which are part of integrations

    • run jobs which are for specific technologies

To clone a production controller to a test controller. Ensure that you have enabled the Quiet start. Take a backup of your ‘production’ controller using the CloudBees Backup plugin.

Omit the secret.key and license.xml from the backup
  • Create a brand new controller running the same version (same docker image) and startup arguments on your ‘test’ or ‘dev’ operations center.

  • Create a restore job and restore the JENKINS_HOME data by using the instructions these Restoring from the CloudBees Backup Plugin.

    1. Update the configuration for some representative jobs so they do not collide with your production jobs, and run them to ensure they work correctly in the upgraded test instance. You will need to re-created credentials on your test instance.

This test environment can be useful after the upgrade as well, for example if you would like to test installing a new plugin.

Upgrade steps

Before you upgrade

  1. Ensure you have enabled the CloudBees Assurance Program (CAP) to ensure you are upgrading to plugin versions tested by CAP. For more information, please refer to Upgrading plugins with Beekeeper Upgrade Assistant.

  2. Enable the CloudBees Quiet Start Plugin plugin to ensure that when the instance first comes online after the upgrade, builds will not start automatically before you get a chance to upgrade the plugins.

  3. Verify the test environment is functioning correctly after it is upgraded.

  4. Ensure you have taken a backup.

  5. Verify all running builds are complete.

  6. Java 11 is recommended for your Jenkins instance, the operations center, Jenkins controllers, and all agents for Jenkins LTS 2.303.3. For more information, refer to the upgrade guide.

CloudBees strongly recommends that you upgrade to Java 11 as soon as possible. The ability to run new versions of the product on Java 8 will be removed in an upcoming release. Releases that occurred during the support window for Java 8 will continue to run on Java 8.
  1. Upgrade the plugins on the operations center.

    1. Select Manage Jenkins  Beekeeper Upgrade Assistant  CAP Configuration.

    2. Verify that Enroll this instance in the CloudBees Assurance Program is selected.

    3. Verify that Allow automatic upgrades of plugins on restart is selected.

    4. Select Save.

    5. Navigate to Manage Jenkins  Manage Plugins and under the Updates tab, select upgrade all plugins.

    6. Restart the operations center.

      If you are using the CAS plugin and upgrading to version 2.277.1.2 or later, do not upgrade it using the Plugin Manager. It must be upgraded at the same time as the instance. Once the service is stopped, you must download the CAS plugin and manually replace $JENKINS_HOME/plugins/cas-plugin.jpi.
  2. Upgrade the Kubernetes resources and the operations center using Helm. For more information refer to Configuring the operations center on modern platforms using CasC and Azure Kubernetes Service (AKS).

    If you are not already using Helm, CloudBees recommends that you migrate to Helm. For more information, refer to Existing CloudBees CI installations to Helm migration guide.
  3. Upgrade the managed controllers. For instructions, refer to Managing controllers. Or, if you want to upgrade managed controllers in bulk, refer to Managing controllers - Bulk upgrades. Please follow the same plugin upgrade steps you followed for the operations center. In addition:

    1. Take a full backup of each controller before you upgrade.

    2. Navigate to Manage Jenkins  Prepare for Shutdown and wait for any running builds to complete before you upgrade a controller.

      You can also connect a client controller to an operations center on modern platforms. For instructions, refer to Upgrading the operations center and controllers.
  4. After your controllers are upgraded, you must manually upgrade the agent.jar file for any inbound agents that are not configured to upgrade the agent.jar file automatically. For more information, refer to Windows agent offline or unable to connect.

  5. Now that you are running the latest version of CloudBees CI , CloudBees recommends you enable actionable build notifications in GitHub, Bitbucket, and Slack. This generally takes a few minutes and provides enhanced notifications for your developers with no Pipeline changes required. For more information, refer to Enabling actionable build notifications in GitHub and Bitbucket and Setting up actionable build notifications in Slack.

  6. When updating your controllers, CloudBees recommends that you install the Health Advisor by CloudBees plugin, if not already installed. The CloudBees Jenkins Health Advisor automatically identifies issues that could impact the performance, stability, and security of your controller. It also identifies potential issues due to known issues on your controller. You are notified via email when a new problem is found and the email includes links to the solutions for the identified issues. For instructions, refer to Jenkins Health Advisor by CloudBees.

    If you are using the High Availability (HA) feature, the general process for updating the HA operations center or any HA controller is:

    1. Stop both HA nodes.

    2. Upgrade both HA nodes.

    3. Start one of the HA nodes and wait for it to come fully online.

    4. Update the plugins according to CAP recommendations and restart.

    5. Start the other HA node.

Previewing upgrade changes using Helm

To preview the changes that will be applied by the upgrade, helm-diff can be used.

  1. Enter the following command:

    $ helm diff upgrade cloudbees-core cloudbees/cloudbees-core -n cloudbees-core -f example-values.yaml (1)
    1 Replace the examples-value.yaml filename with the name of your values file.

    If helm-diff is not an option, enter the following command instead:

    $ diff -u <(helm get manifest cloudbees-core -n cloudbees-core) <(helm template cloudbees-core cloudbees/cloudbees-core -f examples-value.yaml -n cloudbees-core) (1)
    1 Replace the examples-value.yaml filename with the name of your values file.

Upgrading using Helm

One of the most useful features of Helm is a streamlined upgrade and update process for your CloudBees CI installation. Signed Helm chart verification is an optional step in the CloudBees CI on modern cloud platforms upgrade.

The helm upgrade command allows you to upgrade your CloudBees CI release. This can be new setting, such as an updated OperationCenter.HostName value or a new version of CloudBees CI.

Helm Issue 2948 and NGINX Ingress controller support

If you are using the NGINX installation option, you MUST set the ingress-nginx.Enabled=true value or the NGINX Ingress Controller will be deleted. CloudBees strongly recommends you use a Custom Values file.

For more details around Helm Issue 2948, see Helm Issue 2948 on GitHub.

Using the Helm chart to upgrade CloudBees CI on modern cloud platforms

CloudBees provides signed Helm charts for additional security. If you use Helm 3.8 or later and have migrated to the CloudBees CI on modern cloud platforms OCI registry, you can verify the authenticity of the Helm chart.

Upgrading the CloudBees CI on modern cloud platforms to a signed Helm chart requires you to change the URL of the Helm registry to our new OCI registry located at helm.cloudbees.com. If you have built any automation, scripts, or configuration around your existing Helm chart registries, the URL will need to be updated in those places as well.

You can upgrade CloudBees CI on modern cloud platforms to either the latest unsigned or signed Helm chart:

  • Upgrade CloudBees CI on modern cloud platforms to the latest unsigned Helm chart enter the following command:

    helm upgrade -f example-values.yaml --dry-run --debug cloudbees-core cloudbees/cloudbees-core -n cloudbees-core #
  • Upgrade CloudBees CI on modern cloud platforms to the latest signed Helm chart enter the following commands:

    1. Remove the current registry configuration:

      helm repo remove https://public-charts.artifacts.cloudbees.com/repository/public/
    2. Run helm upgrade:

      helm upgrade -f example-values.yaml --dry-run --debug cloudbees-core oci://helm.cloudbees.com/cloudbees-core -n cloudbees-core
    3. (Optional) Run a Helm chart verification to verify the authentication of the signed Helm chart.

Using a custom values file to upgrade CloudBees CI

CloudBees recommends using a Custom Values File to upgrade CloudBees CI using Helm.

To upgrade your installation using a custom values file:

$ helm upgrade -f example-values.yaml --dry-run --debug cloudbees-core cloudbees/cloudbees-core -n cloudbees-core (1)
1 Run the helm upgrade command with the -f argument to use the examples-value.yaml file. Replace the examples-value.yaml filename with the name of your values file. Adding the --dry-run --debug runs a dry run of the installation without making changes.

The following example upgrade custom values file is based on examples in the cloudbees-examples GitHub repository.

# A helm example values file for standard Kubernetes install. # An ingress-nginx controller is not installed and ssl isn't installed. # Install an ingress-nginx controller ingress-nginx: Enabled: false (1) OperationsCenter: # Set the HostName for the Operation Center HostName: 'cloudbees-core.example.com' (2) Ingress: tls: ## Set this to true in order to enable TLS on the ingress record Enable: false (3) SecretName: core-example-com-tls
1 If you want NGINX Ingress controller installed, change this value to true.
2 Replace cloudbees-core.example.com with your domain name.
3 If you want to enable TLS, set this value to true.
Upgrading on controller namespace

If you are deploying controllers in specific Kubernetes namespaces as per Provisioning a controller in a different namespace than the operations center, then you created a release for each namespace. You will need to run the helm upgrade command on each one of those namespaces:

helm upgrade -f example-controllers-values.yaml ci-controllers-$NAMESPACE cloudbees/cloudbees-core -n $NAMESPACE

Upgrading on OpenShift using Helm

One of the most useful features of Helm is a streamlined upgrade and update process for your CloudBees CI installation.

The helm upgrade command allows you to upgrade your CloudBees CI release. This can be new setting, such as an updated OperationCenter.HostName value or a new version of CloudBees CI.

Using a custom values file to upgrade CloudBees CI

CloudBees recommends using a Custom Values File to upgrade CloudBees CI using Helm.

To upgrade your installation using a custom values file:

$ helm upgrade -f openshift-example-values.yaml --dry-run --debug (1) (2)
1 Run the helm upgrade command with the -f argument to use the openshift-examples-value.yaml file. Replace the openshift-examples-value.yaml filename with the name of your values file.
2 Adding the --dry-run --debug runs a dry run of the installation without making changes.

The following example upgrade custom values file is based on examples in the cloudbees-examples GitHub repository.

OperationsCenter: # Set the HostName for the Operation Center HostName: 'cloudbees-core.example.com' (1) Route: tls: ## Set this to true in order to enable TLS on the ingress record Enable: false (2) SecretName: core-example-com-tls Project: name: myproject (3)
1 Replace cloudbees-core.example.com with your domain name.
2 If you want to enable TLS, set this value to true.
3 OperationsCenter.Project.name is the OpenShift project.

If you are deploying controllers in specific OpenShift namespaces as per Provisioning a controller in a different namespace than the operations center, then you created a release for each namespace. You will need to run the helm upgrade command on each one of those namespaces:

helm upgrade -f example-controllers-values.yaml ci-controllers-$NAMESPACE cloudbees/cloudbees-core -n $NAMESPACE

Upgrading without Helm

Before you upgrade

  1. The CloudBees CI YAML has changed, because it is now generated from the CloudBees CI Helm Chart. Both the format and the order of the file have changed. If you intend to use the YAML file as your basis for the upgrade (instead of Helm), you must manually update the field values in cloudbees-core.yml to match the values in your previous installation.

  2. As a best practice, CloudBees recommends backing up your operations center and JENKINS_HOME prior to upgrading. For more information, see the Backup and Restore guide.

Performing the upgrade

If you previously attempted to upgrade CloudBees CI and got an error message reading The StatefulSet "cjoc" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden, follow this upgrade process to resolve the issue.

To upgrade CloudBees CI:

  1. Download the installer from the Downloads site.

  2. In a local directory on your workstation, unpack the installer:

    $ tar xvf name-of-installer.tgz
  3. Modify the extracted cloudbees-core.yml to match the values of your previous installation. There are two options for this step:

    1. Use Helm to create a custom YAML file for your installation

    2. Manually edit cloudbees-core.yml values to match the values from the previous installation (IMPORTANT: the YAML file format has changed.

  4. Delete the cjoc statefulset (this does not remove the associated volume or data):

    $ kubectl delete sts cjoc
  5. Confirm that the operations center statefulset has been deleted:

    $ kubectl get sts cjoc Error from server (NotFound): statefulsets.apps "cjoc" not found (1)
    1 This error confirms that the cjoc statefulset has been deleted.
  6. Run the installer:

    $ kubectl apply -f cloudbees-core.yml
  7. Check the status of the operations center rollout:

    $ kubectl rollout status sts cjoc
  8. Once the new operations center version has been rolled out, sign in to operations center and upgrade your managed controllers.

Restoring from a failed upgrade

If you attempted to upgrade the operations center, but the upgrade failed, you can restore the previous version.

To restore the previous version:

  1. Use helm history to find the revision number you would like to restore to. For example, helm history cloudbees-ci.

  2. Use helm rollback to roll back to the desired revision. For example, helm rollback cloudbees-ci 4.

  3. Scale the cjoc stateful set to 0 replicas and restore the persistent volume data from the backup of the data that was taken before you performed the upgrade, using either your cloud provider’s data restoration features or using Backup and restore on Kubernetes.

If your controller was upgraded and you need to restore the previous version, complete the following steps:

  1. Ensure you are signed in to the operations center as a user with the Administer permission.

  2. From the operations center dashboard, select the down arrow to the right of your controller’s name, select Manage, and then select Stop to stop the controller.

  3. From the operations center dashboard, select the down arrow to the right of your controller’s name, and then select Configure.

  4. From the controller configuration screen, change the Docker image to the previous version, clear Provision and start on save, and then select Save.

  5. Restore the controller’s persistent volume data from the backup of the data that was taken before you performed the upgrade, using either your cloud provider’s data restoration features or using Backup and restore on Kubernetes.

  6. From the operations center dashboard, select the down arrow to the right of your controller’s name, and then select Manage then Start to start the controller.

For additional guidance, please contact CloudBees Support.

In August 2020, the Jenkins project voted to replace the term master with controller. We have taken a pragmatic approach to cleaning these up, ensuring the least amount of downstream impact as possible. CloudBees is committed to ensuring a culture and environment of inclusiveness and acceptance - this includes ensuring the changes are not just cosmetic ones, but pervasive. As this change happens, please note that the term master has been replaced through the latest versions of the CloudBees documentation with controller (as in managed controller, client controller, team controller) except when still used in the UI or in code.