Upgrading CloudBees CI on modern cloud platforms

This guide describes the upgrade process for CloudBees CI on Kubernetes installations.

Helm is the preferred method of managing CloudBees CI on modern cloud platforms. CloudBees provides a Helm chart for CloudBees CI on modern cloud platforms, and recommends migrating existing installations to use the Helm chart.

For instructions on how to migrate your non-Helm installation to Helm, see Migrating existing Kubernetes CloudBees CI installations to Helm.
If you were using Helm 2 to manage your CloudBees CI installation, it has been deprecated. You must migrate to Helm 3. See Migrating from Helm 2 to Helm 3.

The version of operations center must always be more recent or as old as the version of the Managed Masters that are connected to the operations center (operations center >= Managed Master version).

CloudBees recommends that you upgrade your plugins before upgrading a Jenkins instance. For instructions on upgrading plugins, see Upgrading plugins from the Plugin Manager. Run tests after upgrading the plugins to ensure that the upgrades are functioning as expected.

Previewing upgrade changes using Helm

To preview the changes that will be applied by the upgrade, helm-diff can be used.

Enter the following command:

$ helm diff upgrade cloudbees-core cloudbees/cloudbees-core -n cloudbees-core -f example-values.yaml (1)

1	Replace the `examples-value.yaml` filename with the name of your values file.

If helm-diff is not an option, enter the following command instead:

$ diff -u <(helm get manifest cloudbees-core -n cloudbees-core) <(helm template cloudbees-core cloudbees/cloudbees-core -f examples-value.yaml -n cloudbees-core) (1)

1	Replace the `examples-value.yaml` filename with the name of your values file.

Upgrading using Helm

One of the most useful features of Helm is a streamlined upgrade and update process for your CloudBees CI installation.

The helm upgrade command allows you to upgrade your CloudBees CI release. This can be new setting, such as an updated OperationCenter.HostName value or a new version of CloudBees CI.

Helm Issue 2948 and NGINX Ingress controller support

If you are using the NGINX installation option, you MUST set the ingress-nginx.Enabled=true value or the NGINX Ingress Controller will be deleted. CloudBees strongly recommends you use a Custom Values file.

For more details around Helm Issue 2948, see Helm Issue 2948 on GitHub.

Using a custom values file to upgrade CloudBees CI

CloudBees recommends using a Custom Values File to upgrade CloudBees CI using Helm.

To upgrade your installation using a custom values file:

$ helm upgrade -f example-values.yaml --dry-run --debug cloudbees-core cloudbees/cloudbees-core -n cloudbees-core (1)

1	Run the `helm upgrade` command with the `-f` argument to use the `examples-value.yaml` file. Replace the `examples-value.yaml` filename with the name of your values file. Adding the `--dry-run --debug` runs a dry run of the installation without making changes.

The following example upgrade custom values file is based on examples in the cloudbees-examples GitHub repository.

# A helm example values file for standard Kubernetes install.
# An ingress-nginx controller is not installed and ssl isn't installed.
# Install an ingress-nginx controller
ingress-nginx:
  Enabled: false (1)

OperationsCenter:
  # Set the HostName for the Operation Center
  HostName: 'cloudbees-core.example.com' (2)
  Ingress:
    tls:
    ## Set this to true in order to enable TLS on the ingress record
      Enable: false (3)
      SecretName: core-example-com-tls

1	If you want NGINX Ingress controller installed, change this value to `true`.
2	Replace `cloudbees-core.example.com` with your domain name.
3	If you want to enable TLS, set this value to `true`.

If you are deploying controllers in specific Kubernetes namespaces, then you created a release for each namespace. You will need to run the helm upgrade command on each one of those namespaces:

helm upgrade ci-masters-$NAMESPACE cloudbees/cloudbees-core --namespace $NAMESPACE --reuse-values

Upgrading on OpenShift using Helm

One of the most useful features of Helm is a streamlined upgrade and update process for your CloudBees CI installation.

The helm upgrade command allows you to upgrade your CloudBees CI release. This can be new setting, such as an updated OperationCenter.HostName value or a new version of CloudBees CI.

Using a custom values file to upgrade CloudBees CI

CloudBees recommends using a Custom Values File to upgrade CloudBees CI using Helm.

To upgrade your installation using a custom values file:

$ helm upgrade -f openshift-example-values.yaml --dry-run --debug (1) (2)

1	Run the `helm upgrade` command with the `-f` argument to use the `openshift-examples-value.yaml` file. Replace the `openshift-examples-value.yaml` filename with the name of your values file.
2	Adding the `--dry-run --debug` runs a dry run of the installation without making changes.

The following example upgrade custom values file is based on examples in the cloudbees-examples GitHub repository.

OperationsCenter:
  # Set the HostName for the Operation Center
  HostName: 'cloudbees-core.example.com' (1)
  Route:
    tls:
    ## Set this to true in order to enable TLS on the ingress record
      Enable: false (2)
      SecretName: core-example-com-tls
  Project:
    name: myproject (3)

1	Replace `cloudbees-core.example.com` with your domain name.
2	If you want to enable TLS, set this value to `true`.
3	`OperationsCenter.Project.name` is the OpenShift project.

If you are deploying controllers in specific OpenShift namespaces, then you created a release for each namespace. You will need to run the helm upgrade command on each one of those namespaces:

helm upgrade ci-masters-$NAMESPACE cloudbees/cloudbees-core --namespace $NAMESPACE --reuse-values

Upgrading without Helm

Before you upgrade

The CloudBees CI YAML has changed, because it is now generated from the CloudBees CI Helm Chart. Both the format and the order of the file have changed. If you intend to use the YAML file as your basis for the upgrade (instead of Helm), you must manually update the field values in cloudbees-core.yml to match the values in your previous installation.
As a best practice, CloudBees recommends backing up your operations center and JENKINS_HOME prior to upgrading. For more information, see the Backup and Restore guide.

Performing the upgrade

If you previously attempted to upgrade CloudBees CI and got an error message reading The StatefulSet "cjoc" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden, follow this upgrade process to resolve the issue.

To upgrade CloudBees CI:

Download the installer from the Downloads site.
In a local directory on your workstation, unpack the installer:
```
$ tar xvf name-of-installer.tgz
```
Modify the extracted cloudbees-core.yml to match the values of your previous installation. There are two options for this step:
1. Use Helm to create a custom YAML file for your installation
2. Manually edit cloudbees-core.yml values to match the values from the previous installation (IMPORTANT: the YAML file format has changed.
Delete the cjoc statefulset (this does not remove the associated volume or data):
```
$ kubectl delete sts cjoc
```
Confirm that the operations center statefulset has been deleted:
```
$ kubectl get sts cjoc
Error from server (NotFound): statefulsets.apps "cjoc" not found (1)
```
1 This error confirms that the cjoc statefulset has been deleted.
Run the installer:
```
$ kubectl apply -f cloudbees-core.yml
```
Check the status of the operations center rollout:
```
$ kubectl rollout status sts cjoc
```
Once the new operations center version has been rolled out, sign in to operations center and upgrade your Managed Masters.