Upgrade CloudBees CD/RO on Kubernetes

Regularly upgrading to newer versions of CloudBees CD/RO on Kubernetes is encouraged to:

Get the newest CloudBees CD/RO features, security updates, and known issue fixes.
Ensure the longevity of CloudBees CD/RO support for the corresponding Kubernetes release.
Reduce the overall integration effort when older versions reach end-of-life and upgrading may be required.

From a procedural standpoint, upgrading CloudBees CD/RO within Kubernetes is not difficult and may be as simple as updating a few files and running a few commands. However, upgrading may also involve several planning, migration, and integration steps that are specific to your project, your Helm chart values, and environment.

Depending on the versions you are upgrading from and to, features may be added, changed or deprecated that affect your project. Between releases, requirements, supported third-party integrations/plugins, and other key aspects of CloudBees CD/RO may also change. This may mean, upgrading from much older versions to much newer ones may be impractical or impossible without intermediate CloudBees CD/RO release integration.

CloudBees strongly suggests you perform adequate and thorough testing in a development environment before releasing any version upgrades into your production environment.

This section describes upgrading CloudBees CD/RO as a complete component. For specific update instructions:

To update your Helm charts without changing CloudBees CD/RO versions, refer to Update CloudBees CD/RO production server and CloudBees CD/RO agent installation and update.
For Helm chart values for specific components, refer to Configure Helm charts.

Getting started

Before you start your CloudBees CD/RO upgrade, there are several steps you can take to help ease the process:

Ensure the CloudBees CD/RO version you are upgrading to supports the Kubernetes version your platform and database(s) are based on. For more information on which Kubernetes versions are supported by each CloudBees CD/RO release, refer to Supported platforms for CloudBees CD/RO on Kubernetes.
Ensure your Kubernetes platform meets the requirements of the CloudBees CD/RO version you are upgrading to. For more information on CloudBees CD/RO technical requirements, refer to Kubernetes cluster and storage requirements.
Review the CloudBees CD/RO release notes of the version you are upgrading to, and any intermediate releases, to ensure you understand major changes that may have occurred between releases. For more information, refer to CloudBees CD/RO release notes.
If your product relies heavily on specific plugins, review the plugin versions described in the Bundled plugin report for the upgrade release, found in the CloudBees CD/RO release notes. You may also need to review intermediate releases of the plugin(s) to ensure continuity of desired/critical features between releases.
Always make backups of your project-specific resources prior to upgrading. These backups may be vital to recovering your data if critical errors occur during upgrades. For more information, refer to Back up CloudBees CD/RO on Kubernetes.
Always perform adequate and thorough testing in a development environment before releasing any version upgrades into your production environment.
If you run into issues during your upgrade, and need to restore CloudBees CD/RO, refer to Restore CloudBees CD/RO on Kubernetes.

You can integrate CloudBees CD/RO with various Kubernetes platform-specific providers. Some platform-specific integrations, such as OpenShift, have requirements and prerequisites that must be met to correctly install, operate, and upgrade CloudBees CD/RO.

Some requirements must be made in CloudBees CD/RO server and agent values files, while others are made within the platform configuration prior to installing CloudBees CD/RO. For more information on platform-specific integrations, refer to Kubernetes platform-specific configurations.

Additionally, CloudBees CD/RO documentation includes numerous how-tos for Kubernetes project-specific configurations. For help with CloudBees CD/RO on Kubernetes project-specific configurations, refer to Kubernetes configuration options.

How do I know my version is authentic

When searching online, you may find open-source versions of CloudBees CD/RO resources, such as Helm charts and images. Though CloudBees strongly supports the open-source community, it is critically recommended you only install resources in your environment that you fully trust and can verify. CloudBees official resources can be found at:

You can also verify CloudBees resources:

To find more information on verifying Helm charts, refer to Verify Helm charts.
To find more information on verifying Docker images, refer to Verify Docker images.

How does Kubernetes know which version to install

Like most things Kubernetes, the version of CloudBees CD/RO can be defined in the corresponding Helm chart during installation. By default, the values.yaml for a release uses images.registry to know where to retrieve the CloudBees CD/RO image and images.tag to know which image to install. These fields come preconfigured in the default CloudBees Helm charts for each release, as shown in the following example:

images:
  # Image registry to pull the images from.
  # E.g., registry: "123456789012.dkr.ecr.us-east-1.amazonaws.com"
  registry: "docker.io/cloudbees"

  # Image tag of the image to pull
  tag: "10.11.0.159109_3.2.31_20221212"▼

How do I install a specific Helm chart version

You can specify which CloudBees CD/RO default Helm chart to install using helm install with the --version option. When running this command, ensure you are using the version of the Helm chart you want to install and not the CloudBees CD/RO release version. To install a specific Helm chart version:

If you have not done so, add the CloudBees repo:

helm repo add cloudbees https://public-charts.artifacts.cloudbees.com/repository/public/▼

Update the repo and search for available charts:
```
helm repo update
helm search repo cloudbees▼
```

To install a version specific chart, run:

For CloudBees CD/RO flow servers:

helm install <my-releasename> cloudbees/cloudbees-flow \
      --version <Helm-chart-version-to-install>▼

For CloudBees CD/RO flow agents:

helm install <my-releasename> cloudbees/cloudbees-flow-agent \
      --version <Helm-chart-version-to-install>▼

How do I install my project-specific image

CloudBees CD/RO is highly customizable, and depending on your needs, you may not be able to use the default images without customizing them. Conveniently, you can create your own CloudBees CD/RO images based on the default images, store them in your own image host, and install them by configuring your information in a project-specific myvalues.yaml, as shown in the following example:

images:
  # Image registry to pull the images from.
  # E.g., registry: "123456789012.dkr.ecr.us-east-1.amazonaws.com"
  registry: "<your-image-host>"

  # Image tag of the image to pull
  tag: "<your-image-tag>"▼

You can use helm install with the -f option to specify a project-specific values file (myvalues.yaml) during installation. For example:

helm install <my-releasename> cloudbees/cloudbees-flow \
      -f <path/to/myvalues.yaml> \
      --namespace <my-namespace>▼

Upgrade CloudBees CD/RO servers on Kubernetes

For helpful planning considerations, refer to Getting started. If you only want to update server values without upgrading to a new CloudBees CD/RO version, refer to Update CloudBees CD/RO production server.

Before you start:

Refer to CloudBees Helm charts to find the latest available Helm chart version.
Review the CloudBees CD/RO release notes for the latest version, and any intermediate releases, to ensure you understand major changes that may have occurred between releases. For more information, refer to CloudBees CD/RO release notes.
Review the release notes for plugins bundled with the latest version for any project-critical changes. Refer to the Bundled plugin report in the CloudBees CD/RO release notes of the latest version. Some of your plugins may have had several releases since your last integration. Refer the plugin’s release notes for a summary of changes.
Download the latest version’s Helm chart from the CloudBees Nexus repo and compare its changes with your existing values file. Also note its image.tag, you will need this to upgrade the CloudBees CD/RO server image.
Create backups of your CloudBees CD/RO environment. For more information, refer to Back up CloudBees CD/RO on Kubernetes.

Before upgrading from Kubernetes v1.21 or earlier to v1.22 or later

If you are upgrading from Kubernetes v1.21 or earlier, the Kubernetes API underwent an update between v1.21 and v1.22 that included migrating from nginx-ingress to ingress-nginx. For more information, refer to Kubernetes API and Feature Removals In v1.22: Here’s What You Need To Know. For a complete list of Kubernetes APIs that were deprecated in v1.22, refer to the Kubernetes Deprecated API Migration Guide.

To upgrade from Kubernetes v1.21 or lower to v1.22 or higher, you must configure your installation to use ingress-nginx in your Kubernetes v1.21 or earlier values file before attempting to upgrade to v1.22 or later. To do so:

If you do not have a values file that contains the ingress-nginx table:
1. Visit SonaType Nexus, find the latest release of cloudbees-flow, and download the package.
2. Open the package and, in the top-level of the directory, locate the values.yaml.
3. In the values.yaml, locate and copy the ingress-nginx table from the Flow ingress configuration section into your myvalues.yaml.
Ensure the following values are configured in your Kubernetes v1.21 or earlier myvalues.yaml:
- nginx-ingress.enabled=false
- ingress-nginx.enabled=true
In your myvalues.yaml, update any ingress-nginx values as needed by your project.
When performing the upgrade, ensure your values file contains the configured ingress-nginx values.

You can now follow the instructions found in Upgrading your CloudBees CD/RO servers to complete your upgrade.

Troubleshooting upgrades from Kubernetes v1.21 or earlier to v1.22 or later

If you already attempted to upgrade from Kubernetes v1.21 or earlier to v1.22 or later before completing the change from nginx-ingress to ingress-nginx, you may be receiving an error message such as:

unable to recognize "": no matches for kind "xxxxx" in version "xxxxx"▼

This could be the result of CloudBees CD/RO Helm charts with the nginx-ingress third-party chart containing the deprecated rbac.authorization.k8s.io/v1beta1 API, or other project-specific deprecated Kubernetes APIs, in your deployment.

The following instructions explain how to fix this issue. However, as a prerequisite, you must have the following tools installed to follow these steps:

helm
kubectl
gunzip
jq

Once you have confirmed you have these tools installed, to resolve deprecated APIs included in your Kubernetes v1.21 or earlier CloudBees CD/RO Helm chart:

Set the following variables for your v1.21 or earlier deployment:

RELEASE_NAMESPACE="<your-release-namespace>"
RELEASE_NAME="<your-release-name>"▼

Get your currently deployed Kubernetes v1.21 or earlier Helm manifest:

HELM_SECRET_NAME=$(kubectl get secret -l \
owner=helm,status=deployed,name=$RELEASE_NAME \
--namespace $RELEASE_NAMESPACE | awk '{print $1}' | grep -v NAME)

echo $HELM_SECRET_NAME▼

Decode the manifest.json:

kubectl get secrets -n $RELEASE_NAMESPACE $HELM_SECRET_NAME -o json \
| jq .data.release -r | base64 --decode | \
base64 --decode | gunzip - > manifest.json▼

Edit the manifest.json using any editor and replace any APIs that were deprecated as of v1.22. Ensure you replace:
- The nginx-ingress API rbac.authorization.k8s.io/v1beta1 with rbac.authorization.k8s.io/v1.
- Any additional APIs deprecated in Kubernetes release v1.22. For more information, refer to Kubernetes Deprecated API Migration Guide.

Encode the manifest.json:

DATA=`cat manifest.json | gzip -c | base64 | base64`▼

Patch the Helm manifest secret for your Kubernetes v1.21 deployment using $DATA:

kubectl patch secret -n $RELEASE_NAMESPACE $HELM_SECRET_NAME \
--type='json' \
-p="[{\"op\":\"replace\",\"path\":\"/data/release\",\"value\":\"$DATA\"}]"▼

You should receive a success message, such as:
```
secret/sh.helm.release.v1.XXXXX.v1 patched▼
```
If not, ensure:
1. You replaced all Kubernetes v1.21 or earlier APIs that were deprecated in v1.22.
2. The values and paths for all commands were correct.

Now that you have replaced the APIs deprecated in v1.22, you can follow the instructions in Upgrading your CloudBees CD/RO servers.

Upgrading your CloudBees CD/RO servers

To upgrade to the new CloudBees CD/RO version:

Make backups of your deployment and database. These backups may be necessary to recover data if critical errors occurs during the upgrade process.

Set the variables for the current release’s name, its namespace, and version you want to upgrade to:

releaseName="<your-current-release>"
namespace="<your-current-release-namespace>"

# Example upgradeVersion value: "2.21.0"
upgradeVersion="<version-of-upgrade-release>"▼

Save the values from your current release to a values file by running:

helm get values $releaseName --namespace $namespace > old-values.yaml▼

Ensure your old-values.yaml contains values for the following parameters:

If you are upgrading from Kubernetes v1.21 or earlier to v1.22 or later, ensure:

You have performed the steps found in Before upgrading from Kubernetes v1.21 or earlier to v1.22 or later.
The ingress-nginx values are present in your old-values.yaml.
nginx-ingress.enabled=false and ingress-nginx.enabled=true are configured in your old-values.yaml.

Failure to do so, will cause your upgrade to fail. If you have already attempted to do so and your upgrade failed, refer to Troubleshooting upgrades from Kubernetes v1.21 or earlier to v1.22 or later.

ingress.host
storage.volumes.serverPlugins.storageClass
storage.volumes.repositoryStorage.storageClass
storage.volumes.analyticsStorage.storageClass
database.externalEndpoint
database.dbPort
database.dbType
database.dbName
database.dbUser
database.dbPassword
flowLicense

flowCredentials.adminPassword

If you have changed your admin password via the CloudBees CD/RO UI, it is critical you also update your Helm chart values or existingSecret with the new password prior to upgrading. Failing to do so, will cause an upgrade error:

ectool error [AccessDenied]: User 'admin' must be local only.▼

analytics.credentials.adminPassword

If your old-values.yaml uses the images.tag, you must update it with the image tag of the new CloudBees CD/RO release you want to upgrade to. For more information, refer to How does Kubernetes know which version to install.

If you do not update the images.tag or update it with an incorrect value, your upgrade will not succeed.

For CloudBees CD/RO v2023.06.0 and later, this step is OPTIONAL.

Scale down the flow deployments to 1 replica each by running:

kubectl scale deployment flow-server -n $namespace --replicas=1
kubectl scale deployment flow-web -n $namespace --replicas=1
kubectl scale deployment flow-repository -n $namespace --replicas=1▼

Copy and save your flow-server passkey:

# Get the flow-server pod name
cdServerPod=$(kubectl get pod -l app=flow-server -o \
jsonpath='{.items[*].metadata.name}' -n $namespace); echo $cdServerPod

# Get the passkey file name
passkey=$(kubectl -n $namespace exec $cdServerPod -- ls /tmp/ | grep passkey); echo $passkey

# Copy the passkey file from the flow-server pod to the local machine
kubectl -n $namespace cp $cdServerPod:/tmp/$passkey ./passkey

# Ensure the passkey file has been copied
ls -l | grep passkey▼

Copy and store the flow-server keystore files:

# Get the keystore file name
keystore=$(kubectl -n $namespace exec $cdServerPod -- ls /tmp/ | grep keystore); echo $keystore

# Copy the keystore file from flow-server pod to your local machine
kubectl -n $namespace cp $cdServerPod:/tmp/$keystore ./keystore

# Ensure the keystore file has been copied
ls -l | grep keystore▼

To upgrade your release, run:

helm search repo cloudbees
helm pull cloudbees/cloudbees-flow --version $upgradeVersion
tar -xf cloudbees-flow-$upgradeVersion.tgz
helm dep update cloudbees-flow

# If you used a name other than old-values.yaml, use it for --values
helm upgrade $releaseName cloudbees-flow \
--values old-values.yaml \
--namespace $namespace \
# Helm upgrades exit when all tasks are complete, regardless of timeout.
# However, if your database is very large, the timeout may need to be increased to prevent returning a FAILED status.
# For instance: --timeout 10800s
--timeout 4200s▼

If you are installing a CloudBees CD/RO instance with an external database, do not use the --wait option. The flow-server-init-job cannot be started if --wait is used.

The installation may take several minutes. Once the pods are all running, you can track the progress of the CloudBees CD/RO server initialization, via the flow-server-init-job log in the Kubernetes dashboard.

If you run into issues during your upgrade, and need to restore CloudBees CD/RO, refer to Restore to a previous CloudBees CD/RO Helm release.

If you are upgrading from Kubernetes v1.21 or earlier to v1.22 or later, and your upgrade fails an error message such as:

unable to recognize "": no matches for kind "xxxxx" in version "xxxxx"▼

Your Kubernetes v1.21 deployment may have contained APIs deprecated in v1.22. Refer to Troubleshooting upgrades from Kubernetes v1.21 or earlier to v1.22 or later to help resolve this issue.

Bundled plugins updates

Starting with CloudBees CD/RO v2023.08.0, when upgrading your CloudBees CD/RO environment, bundled plugins are only updated or reinstalled if the bundled plugin version changes. This helps decrease upgrade times by skipping reinstalling bundled plugins that had no changes for the CloudBees CD/RO release.

However, after a successful upgrade to v2023.08.0 or later, if you need to reinstall all bundled plugins regardless of whether they have changes, you can force an update to all bundled plugins.

For CloudBees CD/RO v2023.06.0 or earlier, the following steps are not applicable. For each CloudBees CD/RO v2023.06.0 or earlier Kubernetes upgrade, all bundled plugins are reinstalled automatically.

Prerequisites to force-update bundled plugins

The following prerequisites are required to force all bundled plugins to reinstall:

You must successfully upgrade your current CloudBees CD/RO version to v2023.08.0 or later.
You must have CloudBees CD/RO administrator permissions and password for the flow-server instances where the bundled plugin updates should occur.

Force-update bundled plugins

After you have successfully upgraded your current CloudBees CD/RO version to v2023.08.0 or later, perform the following steps to force all bundled plugins to reinstall:

Find the pod name of the flow-server in your namespace:

kubectl get pods --namespace <your-namespace> -l app=flow-server▼

Use kubectl exec to access the flow-server pod and load a bash shell.

kubectl exec --namespace <your-namespace> -ti <pod-name-from-previous-command> -c flow-server -- /bin/bash▼

Log into the flow-server instance as a CloudBees CD/RO administrator using ectool.
```
ectool login admin▼
```
Provide your instance CloudBees CD/RO administrator password.
Execute the bundled plugin force-update using CloudBees ectool.
```
ectool reloadSetupScripts --force 1▼
```

If no error message is returned, the command installs all bundled plugins, and will silently exit when complete.

Upgrade CloudBees CD/RO agents on Kubernetes

For helpful planning considerations, refer to Getting started. If you only want to update your agent’s values without upgrading to a new CloudBees CD/RO version, refer to CloudBees CD/RO agent installation and update.

Your CloudBees CD/RO server and agent(s) can have different CloudBees CD/RO versions and still operate together. However, once you have upgraded your CloudBees CD/RO to a new version, you can upgrade your agent(s) for version continuity.

To upgrade your agent(s):

Make backups of your agent(s) deployment and any project-specific resources. These backups may be necessary to recover data if critical errors occurs during the upgrade process.

Set the variables for the current release’s name, its namespace, and version you want to upgrade to:

releaseName="<your-agents-current-release>"
namespace="<cloudbeesflow-server-namespace>"

# Example upgradeVersion value: "2.21.0"
upgradeVersion="<version-of-upgrade-release>"▼

For $namespace, ensure you use the namespace of the cloudbees-flow instance where you want to upgrade the agent. Using the namespace of an incorrect instance will result in the agent being upgraded for that instance instead of your intended target.

Save the values from your current release to a values file by running:
```
helm get values $releaseName --namespace $namespace > old-agent-values.yaml▼
```
If you used the --set-file customScript option to install third-party tools with your agent, ensure they are included in the old-agent-values.yaml under the customScript tag.
Visit CloudBees Nexus repo, find the release of cloudbees-flow-agent for your upgrade, and download the package.
In the cloudbees-flow-agent package, locate and open the values.yaml.
In the values.yaml, find and copy the agent’s new release image from images.tag.
In your old-values.yaml, replace the images.tag with the agent’s new release image tag and save the file.

To upgrade your release, run:

helm search repo cloudbees
helm pull cloudbees/cloudbees-flow-agent --version $upgradeVersion
tar -xf cloudbees-flow-agent-$upgradeVersion.tgz
helm dep update cloudbees-flow-agent

# If you used a name other than old-agent-values.yaml, use it for --values
helm upgrade $releaseName cloudbees-flow-agent \
--values old-agent-values.yaml \
--namespace $namespace \
--timeout 4200s▼

If you need to add or update third-party tools via script with your agent upgrade, use the following option:

# Optional if you need to add or update third-party tools:
      --set-file customScript=<customScript-file>▼

For more information integrating third-party tools with your agent, refer to Configure agent third-party tools.

The installation may take several minutes. Once the pods are all running, you can track the progress of the CloudBees CD/RO agent initialization via the Kubernetes dashboard.