Upgrading CloudBees CI on modern cloud platforms

This guide describes the upgrade process for CloudBees CI on Kubernetes installations.

Helm is the preferred method of managing CloudBees CI on modern cloud platforms. CloudBees provides a Helm chart for CloudBees CI on modern cloud platforms, and recommends migrating existing installations to use the Helm chart.
The version of operations center must always be more recent or as old as the version of the managed controllers that are connected to the operations center (operations center >= managed controller version).
CloudBees recommends that you upgrade your plugins before upgrading a Jenkins instance. For instructions on upgrading plugins, see Upgrading plugins from the Plugin Manager. Run tests after upgrading the plugins to ensure that the upgrades are functioning as expected.

Key principles to follow when upgrading from one version to the other

Change one thing at a time

Limit the changes you carry out to one area at a time. For example, do not plan to make infrastructure changes such as hardware upgrades, networking changes, OS package upgrades at the same time as upgrading Jenkins. This will help in identifying and isolating the root cause of any issues encountered.

Backup strategy

  1. Ensure that you have regular backups of your JENKINS_HOME for your operations center, and all controllers. We recommend using the CloudBees Backup Plugin to automate this. controllers that have CloudBees Backup Plugin 3.38 or later installed have support for the Restore job type. To restore an operations center when using CloudBees CI on traditional platforms follow the steps for Restoring manually. If you are using CloudBees CI on modern cloud platforms, follow the steps to Backup and restore on Kubernetes. The backups should be stored at an offsite location such as cloud storage or at a minimum on separate hardware from your CloudBees CI.

  2. Take another backup immediately before you start the upgrade.

  3. Test that this backup is valid as we use it to recover from a failure if we encounter a severe issue during the production upgrade. A great way to test the validity of these backups is with a test environment. Another reason for testing the builds is that after an upgrade of the Jenkins core version or plugins, you cannot directly downgrade the war file and plugins, the only way to 'roll back' is to restore the JENKINS_HOME backup. To validate the integrity of the backup it is essential to extract the backup to another filesystem, and verify the checksums using:

  ----
  find $JENKINS_HOME -type f | sort | xargs md5sum >~/checksum.log.backup
  ----

Run this for both the live and backup data, and compare the checksum.log.* between the two.

Test strategy

  1. We recommend you have a separate test environment with a similar setup to your production environment, with representative jobs. If you are creating a test environment and require a license please send us the instance ID of your test instance and we will send you a test instance license.

  2. We recommend having jobs on your test controller that are representative of jobs on your production controller, so that testing the upgrade, uses the key plugins to your workflow, and the interactions with any external systems (SCM, artifact repositories, bug trackers, cloud automation). It is a best practice to build some jobs which are a good subset of representative jobs on this test cluster. These dedicated jobs represent your production jobs, but without working on your real data. We recommend talking to your development groups to find jobs that cover the main functionality. Here are some recommendations:

    • test your end-to-end complex pipeline jobs

    • test those jobs that utilize critical plugins

    • run jobs which are part of integrations

    • run jobs which are for specific technologies

To clone a production controller to a test controller. Ensure that you have enabled the Quiet start. Take a backup of your ‘production’ controller using the CloudBees Backup plugin.

Omit the secret.key and license.xml from the backup
  • Create a brand new controller running the same version (same docker image) and startup arguments on your ‘test’ or ‘dev’ operations center.

  • Create a restore job and restore the JENKINS_HOME data by using the instructions these Restoring from the CloudBees Backup Plugin.

    1. Update the configuration for some representative jobs so they do not collide with your production jobs, and run them to ensure they work correctly in the upgraded test instance. You will need to re-created credentials on your test instance.

This test environment can be useful after the upgrade as well, for example if you would like to test installing a new plugin. Unresolved include directive in modules/cloud-upgrade-guide/pages/index.adoc - include::cloudbees-common::partial$upgrade-guide/key_upgrade_steps.adoc[]

Previewing upgrade changes using Helm

To preview the changes that will be applied by the upgrade, helm-diff can be used.

  1. Enter the following command:

    $ helm diff upgrade cloudbees-core cloudbees/cloudbees-core -n cloudbees-core -f example-values.yaml (1)
    1Replace the examples-value.yaml filename with the name of your values file.

    If helm-diff is not an option, enter the following command instead:

    $ diff -u <(helm get manifest cloudbees-core -n cloudbees-core) <(helm template cloudbees-core cloudbees/cloudbees-core -f examples-value.yaml -n cloudbees-core) (1)
    1Replace the examples-value.yaml filename with the name of your values file.

Upgrading using Helm

One of the most useful features of Helm is a streamlined upgrade and update process for your CloudBees CI installation.

The helm upgrade command allows you to upgrade your CloudBees CI release. This can be new setting, such as an updated OperationCenter.HostName value or a new version of CloudBees CI.

Helm Issue 2948 and NGINX Ingress controller support

If you are using the NGINX installation option, you MUST set the ingress-nginx.Enabled=true value or the NGINX Ingress Controller will be deleted. CloudBees strongly recommends you use a Custom Values file.

For more details around Helm Issue 2948, see Helm Issue 2948 on GitHub.

Using a custom values file to upgrade CloudBees CI

CloudBees recommends using a Custom Values File to upgrade CloudBees CI using Helm.

To upgrade your installation using a custom values file:

$ helm upgrade -f example-values.yaml --dry-run --debug cloudbees-core cloudbees/cloudbees-core -n cloudbees-core (1)
1Run the helm upgrade command with the -f argument to use the examples-value.yaml file. Replace the examples-value.yaml filename with the name of your values file. Adding the --dry-run --debug runs a dry run of the installation without making changes.

The following example upgrade custom values file is based on examples in the cloudbees-examples GitHub repository.

# A helm example values file for standard Kubernetes install.
# An ingress-nginx controller is not installed and ssl isn't installed.
# Install an ingress-nginx controller
ingress-nginx:
  Enabled: false (1)

OperationsCenter:
  # Set the HostName for the Operation Center
  HostName: 'cloudbees-core.example.com' (2)
  Ingress:
    tls:
    ## Set this to true in order to enable TLS on the ingress record
      Enable: false (3)
      SecretName: core-example-com-tls
1If you want NGINX Ingress controller installed, change this value to true.
2Replace cloudbees-core.example.com with your domain name.
3If you want to enable TLS, set this value to true.

If you are deploying controllers in specific Kubernetes namespaces, then you created a release for each namespace. You will need to run the helm upgrade command on each one of those namespaces:

helm upgrade ci-masters-$NAMESPACE cloudbees/cloudbees-core --namespace $NAMESPACE --reuse-values

Upgrading on OpenShift using Helm

One of the most useful features of Helm is a streamlined upgrade and update process for your CloudBees CI installation.

The helm upgrade command allows you to upgrade your CloudBees CI release. This can be new setting, such as an updated OperationCenter.HostName value or a new version of CloudBees CI.

Using a custom values file to upgrade CloudBees CI

CloudBees recommends using a Custom Values File to upgrade CloudBees CI using Helm.

To upgrade your installation using a custom values file:

$ helm upgrade -f openshift-example-values.yaml --dry-run --debug (1) (2)
1Run the helm upgrade command with the -f argument to use the openshift-examples-value.yaml file. Replace the openshift-examples-value.yaml filename with the name of your values file.
2Adding the --dry-run --debug runs a dry run of the installation without making changes.

The following example upgrade custom values file is based on examples in the cloudbees-examples GitHub repository.

OperationsCenter:
  # Set the HostName for the Operation Center
  HostName: 'cloudbees-core.example.com' (1)
  Route:
    tls:
    ## Set this to true in order to enable TLS on the ingress record
      Enable: false (2)
      SecretName: core-example-com-tls
  Project:
    name: myproject (3)
1Replace cloudbees-core.example.com with your domain name.
2If you want to enable TLS, set this value to true.
3OperationsCenter.Project.name is the OpenShift project.

If you are deploying controllers in specific OpenShift namespaces, then you created a release for each namespace. You will need to run the helm upgrade command on each one of those namespaces:

helm upgrade ci-masters-$NAMESPACE cloudbees/cloudbees-core --namespace $NAMESPACE --reuse-values

Upgrading without Helm

Before you upgrade

  1. The CloudBees CI YAML has changed, because it is now generated from the CloudBees CI Helm Chart. Both the format and the order of the file have changed. If you intend to use the YAML file as your basis for the upgrade (instead of Helm), you must manually update the field values in cloudbees-core.yml to match the values in your previous installation.

  2. As a best practice, CloudBees recommends backing up your operations center and JENKINS_HOME prior to upgrading. For more information, see the Backup and Restore guide.

Performing the upgrade

If you previously attempted to upgrade CloudBees CI and got an error message reading The StatefulSet "cjoc" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden, follow this upgrade process to resolve the issue.

To upgrade CloudBees CI:

  1. Download the installer from the Downloads site.

  2. In a local directory on your workstation, unpack the installer:

    $ tar xvf name-of-installer.tgz
  3. Modify the extracted cloudbees-core.yml to match the values of your previous installation. There are two options for this step:

    1. Use Helm to create a custom YAML file for your installation

    2. Manually edit cloudbees-core.yml values to match the values from the previous installation (IMPORTANT: the YAML file format has changed.

  4. Delete the cjoc statefulset (this does not remove the associated volume or data):

    $ kubectl delete sts cjoc
  5. Confirm that the operations center statefulset has been deleted:

    $ kubectl get sts cjoc
    Error from server (NotFound): statefulsets.apps "cjoc" not found (1)
    1This error confirms that the cjoc statefulset has been deleted.
  6. Run the installer:

    $ kubectl apply -f cloudbees-core.yml
  7. Check the status of the operations center rollout:

    $ kubectl rollout status sts cjoc
  8. Once the new operations center version has been rolled out, sign in to operations center and upgrade your managed controllers.

In August 2020, the Jenkins project voted to replace the term master with controller. We have taken a pragmatic approach to cleaning these up, ensuring the least amount of downstream impact as possible. CloudBees is committed to ensuring a culture and environment of inclusiveness and acceptance - this includes ensuring the changes are not just cosmetic ones, but pervasive. As this change happens, please note that the term master has been replaced through the latest versions of the CloudBees documentation with controller (as in managed controller, client controller, team controller) except when still used in the UI or in code.