Upgrading CloudBees CI on modern cloud platforms

This guide describes the upgrade process for CloudBees CI on Kubernetes installations. CloudBees recommends that you always upgrade to the latest version of your CloudBees product, ensuring wider lifecycle support coverage, plus more security patches added via the CloudBees Security Advisory, resolved issues, and new features.

Helm is the preferred method of managing CloudBees CI on modern cloud platforms. CloudBees provides a Helm chart for CloudBees CI on modern cloud platforms, and recommends migrating existing installations to use the Helm chart.

For instructions on how to migrate your non-Helm installation to Helm, see Migrating existing Kubernetes CloudBees CI installations to Helm.
If you were using Helm 2 to manage your CloudBees CI installation, it has been deprecated. You must migrate to Helm 3. See Migrating from Helm 2 to Helm 3.

For controllers that are in your environment:
- The operations center must be upgraded first, and then controllers can be upgraded.
- The controller used with the operations center does not have to be at the same version as the operations center. The operations center version must always be more recent or as old as the version of the controllers that are used with the operations center (operations center >= controller version). CloudBees supports upgrading CloudBees CI to a version up to one year later than your current version.

For more information, refer to Version skew between the operations center and controllers.

Choose an upgrade target

CloudBees supports upgrading your CloudBees CI instance to a version up to one year later than your current version, as documented in Upgrading between versions.

For this reason, upgrade your CloudBees CI environment on a regular basis. If your instance is more than a year old, multiple upgrades will be required to upgrade to the latest release.

If running versions older than 2.452.2.3, follow the specific version guidance from Upgrading from CloudBees CI versions older than 2.452.2.3

Key principles to follow when upgrading from one version to the other

Change one thing at a time

Limit the changes you carry out to one area at a time. For example, do not plan to make infrastructure changes such as hardware upgrades, networking changes, OS package upgrades at the same time as upgrading Jenkins. This will help in identifying and isolating the root cause of any issues encountered.

Backup strategy

Ensure that you have regular backups of your JENKINS_HOME for your operations center, and all controllers. We recommend using the CloudBees Backup Plugin to automate this. controllers that have CloudBees Backup Plugin 3.38 or later installed have support for the Restore job type. To restore an operations center when using CloudBees CI on modern cloud platforms follow the steps for Restoring manually. If you are using CloudBees CI on modern cloud platforms, follow the steps to Backup and restore on Kubernetes. The backups should be stored at an offsite location such as cloud storage or at a minimum on separate hardware from your CloudBees CI.
Take another backup immediately before you start the upgrade.
Test that this backup is valid as we use it to recover from a failure if we encounter a severe issue during the production upgrade. A great way to test the validity of these backups is with a test environment. Another reason for testing the builds is that after an upgrade of the Jenkins core version or plugins, you cannot directly downgrade the war file and plugins, the only way to 'roll back' is to restore the JENKINS_HOME backup. To validate the integrity of the backup it is essential to extract the backup to another filesystem, and verify the checksums using:
```
find $JENKINS_HOME -type f -print0 | sort -z | xargs -0 md5sum >~/checksum.log.backup▼
```
Run this for both the live and backup data, and compare the checksum.log.* between the two.

Test strategy

We recommend you have a separate test environment with a similar setup to your production environment, with representative jobs. If you are creating a test environment and require a license please send us the instance ID of your test instance and we will send you a test instance license.
We recommend having jobs on your test controller that are representative of jobs on your production controller, so that testing the upgrade, uses the key plugins to your workflow, and the interactions with any external systems (SCM, artifact repositories, bug trackers, cloud automation). It is a best practice to build some jobs which are a good subset of representative jobs on this test cluster. These dedicated jobs represent your production jobs, but without working on your real data. We recommend talking to your development groups to find jobs that cover the main functionality. Here are some recommendations:
- test your end-to-end complex pipeline jobs
- test those jobs that utilize critical plugins
- run jobs which are part of integrations
- run jobs which are for specific technologies

To clone a production controller to a test controller. Ensure that you have enabled the Quiet start. Take a backup of your ‘production’ controller using the CloudBees Backup plugin.

Omit the secret.key and license.xml from the backup

Create a brand new controller running the same version (same docker image) and startup arguments on your ‘test’ or ‘dev’ operations center.
Create a restore job and restore the JENKINS_HOME data by using the instructions these Restoring from the CloudBees Backup Plugin.
1. Update the configuration for some representative jobs so they do not collide with your production jobs, and run them to ensure they work correctly in the upgraded test instance. You will need to re-created credentials on your test instance.

This test environment can be useful after the upgrade as well, for example if you would like to test installing a new plugin.

Upgrade steps

Before you upgrade

Ensure you have enabled the CloudBees Assurance Program (CAP) to ensure you are upgrading to plugin versions tested by CAP. For more information, please refer to Upgrading plugins with Beekeeper Upgrade Assistant.
Enable the CloudBees Quiet Start plugin to ensure that when the instance first comes online after the upgrade, builds will not start automatically before you get a chance to upgrade the plugins.
Verify the test environment is functioning correctly after it is upgraded.
Ensure you have taken a backup.
Verify all running builds are complete.
Java 11 is recommended for your Jenkins instance, the operations center, Jenkins controllers, and all agents for Jenkins LTS 2.303.3. For more information, refer to the upgrade guide.

CloudBees strongly recommends that you upgrade to Java 11 as soon as possible. The ability to run new versions of the product on Java 8 will be removed in an upcoming release. Releases that occurred during the support window for Java 8 will continue to run on Java 8.

Recommended steps

Upgrade the plugins on the operations center.

Select Manage Jenkins Beekeeper Upgrade Assistant CAP Configuration.
Verify that Enroll this instance in the CloudBees Assurance Program is selected.
Verify that Allow automatic upgrades of plugins on restart is selected.
Select Save.
Navigate to Manage Jenkins Manage Plugins and under the Updates tab, select upgrade all plugins.

Restart the operations center.

If you are using the CAS plugin and upgrading to version 2.277.1.2 or later, do not upgrade it using the Plugin Manager. It must be upgraded at the same time as the instance. Once the service is stopped, you must download the CAS plugin and manually replace $JENKINS_HOME/plugins/cas-plugin.jpi.

Upgrade the Kubernetes resources and the operations center using Helm. For more information refer to Upgrading using Helm.

If you are not already using Helm, CloudBees recommends that you migrate to Helm. For more information, refer to Upgrading CloudBees CI on modern cloud platforms.
Upgrade the managed controllers. For instructions, refer to Manage controllers. Or, if you want to upgrade managed controllers in bulk, refer to Managing controllers - Bulk upgrades. Please follow the same plugin upgrade steps you followed for the operations center. In addition:
1. Take a full backup of each controller before you upgrade.
2. Navigate to Manage Jenkins Prepare for Shutdown and wait for any running builds to complete before you upgrade a controller.
  
  You can also connect a client controller to an operations center on modern platforms. For instructions, refer to Upgrading the operations center and controllers.
After your controllers are upgraded, you must manually upgrade the agent.jar file for any inbound agents that are not configured to upgrade the agent.jar file automatically. For more information, refer to Windows agent offline or unable to connect.
SSH-connected agents upgrade the agent.jar automatically.

You can enable automatic upgrades for inbound agents by Upgrading Windows Service Wrapper.
Now that you are running the latest version of CloudBees CI, CloudBees recommends you enable actionable build notifications in GitHub, Bitbucket, and Slack. This generally takes a few minutes and provides enhanced notifications for your developers with no Pipeline changes required. For more information, refer to Setting up actionable build notifications in Slack.

When updating your controllers, CloudBees recommends that you install the Health Advisor by CloudBees plugin, if not already installed. The CloudBees Jenkins Health Advisor automatically identifies issues that could impact the performance, stability, and security of your controller. It also identifies potential issues due to known issues on your controller. You are notified via email when a new problem is found and the email includes links to the solutions for the identified issues. For instructions, refer to Jenkins Health Advisor by CloudBees.

If you are using the High Availability (active/passive) feature, the general process for updating the operations center or controller is:

Stop both HA nodes.
Upgrade both HA nodes.
Start one of the HA nodes and wait for it to come fully online.
Update the plugins according to CAP recommendations and restart.
Start the other HA node.

Previewing upgrade changes using Helm

To preview the changes that will be applied by the upgrade, helm-diff can be used.

Enter the following command:

$ helm diff upgrade cloudbees-core cloudbees/cloudbees-core -n cloudbees-core -f example-values.yaml(1)▼

1	Replace the `examples-value.yaml` filename with the name of your values file.

If helm-diff is not an option, enter the following command instead:

$ diff -u <(helm get manifest cloudbees-core -n cloudbees-core) <(helm template cloudbees-core cloudbees/cloudbees-core -f examples-value.yaml -n cloudbees-core)(1)▼

1	Replace the `examples-value.yaml` filename with the name of your values file.

Upgrading using Helm

One of the most useful features of Helm is a streamlined upgrade and update process for your CloudBees CI installation. Signed Helm chart verification is an optional step in the CloudBees CI on modern cloud platforms upgrade.

The helm upgrade command allows you to upgrade your CloudBees CI release. This can be new setting, such as an updated OperationCenter.HostName value or a new version of CloudBees CI.

Using the Helm chart to upgrade CloudBees CI on modern cloud platforms

CloudBees provides signed Helm charts for additional security. If you use Helm 3.8 or later and have migrated to the CloudBees CI on modern cloud platforms OCI registry, you can verify the authenticity of the Helm chart.

Upgrading the CloudBees CI on modern cloud platforms to a signed Helm chart requires you to change the URL of the Helm registry to our new OCI registry located at helm.cloudbees.com. If you have built any automation, scripts, or configuration around your existing Helm chart registries, the URL will need to be updated in those places as well.

You can upgrade CloudBees CI on modern cloud platforms to either the latest unsigned or signed Helm chart:

Upgrade CloudBees CI on modern cloud platforms to the latest unsigned Helm chart enter the following command:

helm upgrade -f example-values.yaml --dry-run --debug cloudbees-core cloudbees/cloudbees-core -n cloudbees-core▼

Upgrade CloudBees CI on modern cloud platforms to the latest signed Helm chart enter the following commands:
1. Remove the current registry configuration:
  helm repo remove https://public-charts.artifacts.cloudbees.com/repository/public/
  ▼
2. Run helm upgrade:
  helm upgrade -f example-values.yaml --dry-run --debug cloudbees-core oci://helm.cloudbees.com/cloudbees-core -n cloudbees-core
  ▼
3. (Optional) Run a Helm chart verification to verify the authentication of the signed Helm chart.

Using a custom values file to upgrade CloudBees CI

CloudBees recommends using a Custom Values File to upgrade CloudBees CI using Helm.

To upgrade your installation using a custom values file:

$ helm upgrade -f example-values.yaml --dry-run --debug cloudbees-core cloudbees/cloudbees-core -n cloudbees-core(1)▼

1	Run the `helm upgrade` command with the `-f` argument to use the `examples-value.yaml` file. Replace the `examples-value.yaml` filename with the name of your values file. Adding the `--dry-run --debug` runs a dry run of the installation without making changes.

The following example upgrade custom values file is based on examples in the cloudbees-examples GitHub repository.

# A helm example values file for standard Kubernetes install.
# Embedded NGINX Ingress controller is not installed and SSL isn't installed.
ingress-nginx:
  Enabled: false(1)

OperationsCenter:
  # Set the HostName for the Operation Center
  HostName: 'cloudbees-core.example.com'(2)
  Ingress:
    tls:
    ## Set this to true in order to enable TLS on the ingress record
      Enable: false(3)
      SecretName: core-example-com-tls▼

1	Setting this value to `true` is deprecated. If you want NGINX Ingress controller installed, install it separately.
2	Replace `cloudbees-core.example.com` with your domain name.
3	If you want to enable TLS, set this value to `true`.

Upgrading on controller namespace

If you are deploying controllers in specific Kubernetes namespaces as per Provisioning a controller in a different namespace than the operations center, then you created a release for each namespace. You will need to run the helm upgrade command on each one of those namespaces:

helm upgrade -f example-controllers-values.yaml ci-controllers-$NAMESPACE cloudbees/cloudbees-core -n $NAMESPACE▼

Upgrading on OpenShift using Helm

One of the most useful features of Helm is a streamlined upgrade and update process for your CloudBees CI installation.

The helm upgrade command allows you to upgrade your CloudBees CI release. This can be new setting, such as an updated OperationCenter.HostName value or a new version of CloudBees CI.

Using a custom values file to upgrade CloudBees CI

CloudBees recommends using a Custom Values File to upgrade CloudBees CI using Helm.

To upgrade your installation using a custom values file:

$ helm upgrade -f openshift-example-values.yaml --dry-run --debug(1) (2)▼

1	Run the `helm upgrade` command with the `-f` argument to use the `openshift-examples-value.yaml` file. Replace the `openshift-examples-value.yaml` filename with the name of your values file.
2	Adding the `--dry-run --debug` runs a dry run of the installation without making changes.

The following example upgrade custom values file is based on examples in the cloudbees-examples GitHub repository.

OperationsCenter:
  # Set the HostName for the Operation Center
  HostName: 'cloudbees-core.example.com'(1)
  Route:
    tls:
    ## Set this to true in order to enable TLS on the ingress record
      Enable: false(2)
      SecretName: core-example-com-tls
  Project:
    name: myproject(3)▼

1	Replace `cloudbees-core.example.com` with your domain name.
2	If you want to enable TLS, set this value to `true`.
3	`OperationsCenter.Project.name` is the OpenShift project.

If you are deploying controllers in specific OpenShift namespaces as per Provisioning a controller in a different namespace than the operations center, then you created a release for each namespace. You will need to run the helm upgrade command on each one of those namespaces:

helm upgrade -f example-controllers-values.yaml ci-controllers-$NAMESPACE cloudbees/cloudbees-core -n $NAMESPACE▼

Restoring from a failed upgrade

If you attempted to upgrade the operations center, but the upgrade failed, you can restore the previous version.

To restore the previous version:

Use helm history to find the revision number you would like to restore to. For example, helm history cloudbees-ci.
Use helm rollback to roll back to the desired revision. For example, helm rollback cloudbees-ci 4.
Scale the cjoc stateful set to 0 replicas and restore the persistent volume data from the backup of the data that was taken before you performed the upgrade, using either your cloud provider’s data restoration features or using Backup and restore on Kubernetes.

If your controller was upgraded, and you need to restore the previous version, complete the following steps:

Ensure you are signed in to the operations center as a user with the Administer permission.
From the operations center dashboard, select the down arrow to the right of your controller’s name, select Manage, and then select Stop to stop the controller.
From the operations center dashboard, select the down arrow to the right of your controller’s name, and then select Configure.
From the controller configuration screen, change the Docker image to the previous version, clear Provision and start on save, and then select Save.
Restore the controller’s persistent volume data from the backup of the data that was taken before you performed the upgrade, using either your cloud provider’s data restoration features or using Backup and restore on Kubernetes.
From the operations center dashboard, select the down arrow to the right of your controller’s name, and then select Manage then Start to start the controller.

For additional guidance, please contact CloudBees Support.