# Upgrade CloudBees CD/RO on Kubernetes Upgrade CloudBees CD/RO on Kubernetes ===================================== Regularly upgrading to newer versions of CloudBees CD/RO on Kubernetes is encouraged to: * Get the newest CloudBees CD/RO features, security updates, and known issue fixes. * Ensure the longevity of CloudBees CD/RO support for the corresponding Kubernetes release. * Reduce the overall integration effort when older versions reach end-of-life and upgrading may be required. From a procedural standpoint, upgrading CloudBees CD/RO within Kubernetes is not difficult and may be as simple as updating a few files and running a few commands. However, upgrading may also involve several planning, migration, and integration steps that are specific to your project, your Helm chart values, and environment. Depending on the versions you are upgrading from and to, features may be added, changed or deprecated that affect your project. Between releases, requirements, supported third-party integrations/plugins, and other key aspects of CloudBees CD/RO may also change. This may mean, upgrading from much older versions to much newer ones may be impractical or impossible without intermediate CloudBees CD/RO release integration. CloudBees strongly suggests you perform adequate and thorough testing in a development environment before releasing any version upgrades into your production environment. This section describes upgrading CloudBees CD/RO as a complete component. For specific update instructions: * To update your Helm charts without changing CloudBees CD/RO versions, refer to [Update CloudBees CD/RO production server](/docs/cloudbees-cd/latest/install-k8s/installation#update-production-servers) and [CloudBees CD/RO agent installation and update](/docs/cloudbees-cd/latest/install-k8s/installation#Install-agent). * For Helm chart values for specific components, refer to [Configure Helm charts](/docs/cloudbees-cd/latest/install-k8s/configuration). [](#_getting_started)Getting started ------------------------------------ Before you start your CloudBees CD/RO upgrade, there are several steps you can take to help ease the process: 1. Ensure the CloudBees CD/RO version you are upgrading to supports the Kubernetes version your platform and database(s) are based on. For more information on which Kubernetes versions are supported by each CloudBees CD/RO release, refer to [Supported platforms for CloudBees CD/RO on Kubernetes](/docs/cloudbees-common/latest/supported-platforms/cloudbees-cd-k8s). 2. Ensure your Kubernetes platform meets the requirements of the CloudBees CD/RO version you are upgrading to. For more information on CloudBees CD/RO technical requirements, refer to [Kubernetes cluster and storage requirements](/docs/cloudbees-cd/latest/requirements/k8s-requirements). 3. Review the CloudBees CD/RO release notes of the version you are upgrading to, and any intermediate releases, to ensure you understand major changes that may have occurred between releases. For more information, refer to [CloudBees CD/RO release notes](/docs/cloudbees-cd-release-notes/latest/). 4. If your product relies heavily on specific plugins, review the plugin versions described in the **Bundled plugin report** for the upgrade release, found in the [CloudBees CD/RO release notes](/docs/cloudbees-cd-release-notes/latest/). You may also need to review intermediate releases of the plugin(s) to ensure continuity of desired/critical features between releases. 5. Always make backups of your project-specific resources prior to upgrading. These backups may be vital to recovering your data if critical errors occur during upgrades. For more information, refer to [Back up CloudBees CD/RO on Kubernetes](/docs/cloudbees-cd/latest/maintain-cdro/k8s-server-backups). 6. Always perform adequate and thorough testing in a development environment before releasing any version upgrades into your production environment. 7. If you run into issues during your upgrade, and need to restore CloudBees CD/RO, refer to [Restore CloudBees CD/RO on Kubernetes](/docs/cloudbees-cd/latest/maintain-cdro/k8s-server-restore). You can integrate CloudBees CD/RO with various Kubernetes platform-specific providers. Some platform-specific integrations, such as OpenShift, have requirements and prerequisites that must be met to correctly install, operate, and upgrade CloudBees CD/RO. Some requirements must be made in CloudBees CD/RO server and agent values files, while others are made within the platform configuration prior to installing CloudBees CD/RO. For more information on platform-specific integrations, refer to [Kubernetes platform-specific configurations](/docs/cloudbees-cd/latest/install-k8s/k8s-platform-specific-configurations). Additionally, CloudBees CD/RO documentation includes numerous how-tos for Kubernetes project-specific configurations. For help with CloudBees CD/RO on Kubernetes project-specific configurations, refer to [Kubernetes how-to articles](/docs/cloudbees-cd/latest/k8s-how-to/). ### [](#_how_do_i_know_my_version_is_authentic)How do I know my version is authentic When searching online, you may find open-source versions of CloudBees CD/RO resources, such as Helm charts and images. Though CloudBees strongly supports the open-source community, it is **_critically_** recommended you only install resources in your environment that you fully trust and can verify. CloudBees official resources can be found at: * [CloudBees Helm charts](https://public-charts.artifacts.cloudbees.com/#browse/browse:public) * [CloudBees Docker images](https://hub.docker.com/u/cloudbees) You can also verify CloudBees resources: * To find more information on verifying Helm charts, refer to [Verify Helm charts](/docs/cloudbees-cd/latest/install-k8s/verifying-helm-chart-signatures). * To find more information on verifying Docker images, refer to [Verify Docker images](/docs/cloudbees-cd/latest/install-k8s/verifying-docker-image-signatures). ### [](#_how_does_kubernetes_know_which_version_to_install)How does Kubernetes know which version to install Like most things Kubernetes, the version of CloudBees CD/RO can be defined in the corresponding Helm chart during installation. By default, the `values.yaml` for a release uses `images.registry` to know where to retrieve the CloudBees CD/RO image and `images.tag` to know which image to install. These fields come preconfigured in the default [CloudBees Helm charts](https://public-charts.artifacts.cloudbees.com/#browse/browse:public) for each release, as shown in the following example: images: # Image registry to pull the images from. # E.g., registry: "123456789012.dkr.ecr.us-east-1.amazonaws.com" registry: "docker.io/cloudbees" # Image tag of the image to pull tag: "10.11.0.159109_3.2.31_20221212" ### [](#_how_do_i_install_a_specific_helm_chart_version)How do I install a specific Helm chart version You can specify which CloudBees CD/RO default Helm chart to install using `helm install` with the `--version` option. When running this command, ensure you are using the version of the Helm chart you want to install and not the CloudBees CD/RO release version. To install a specific Helm chart version: 1. If you have not done so, add the CloudBees repo: helm repo add cloudbees https://public-charts.artifacts.cloudbees.com/repository/public/ 2. Update the repo and search for available charts: helm repo update helm search repo cloudbees 3. To install a version specific chart, run: 1. For CloudBees CD/RO flow servers: helm install cloudbees/cloudbees-flow \ --version 2. For CloudBees CD/RO flow agents: helm install cloudbees/cloudbees-flow-agent \ --version #### [](#_how_do_i_install_my_project_specific_image)How do I install my project-specific image CloudBees CD/RO is highly customizable, and depending on your needs, you may not be able to use the default images without customizing them. Conveniently, you can create your own CloudBees CD/RO images based on the default images, store them in your own image host, and install them by configuring your information in a project-specific `myvalues.yaml`, as shown in the following example: images: # Image registry to pull the images from. # E.g., registry: "123456789012.dkr.ecr.us-east-1.amazonaws.com" registry: "" # Image tag of the image to pull tag: "" You can use `helm install` with the `-f` option to specify a project-specific values file (`myvalues.yaml`) during installation. For example: helm install cloudbees/cloudbees-flow \ -f \ --namespace [](#_upgrade_cloudbees_cdro_servers_on_kubernetes)Upgrade CloudBees CD/RO servers on Kubernetes ----------------------------------------------------------------------------------------------- For helpful planning considerations, refer to [Getting started](#_getting_started). If you only want to update server values without upgrading to a new CloudBees CD/RO version, refer to [Update CloudBees CD/RO production server](/docs/cloudbees-cd/latest/install-k8s/installation#update-production-servers). ### [](#_before_you_start)Before you start: * Refer to [CloudBees Helm charts](https://public-charts.artifacts.cloudbees.com/#browse/browse:public) to find the latest available Helm chart version. * Review the CloudBees CD/RO release notes for the latest version, and any intermediate releases, to ensure you understand major changes that may have occurred between releases. For more information, refer to [CloudBees CD/RO release notes](/docs/cloudbees-cd-release-notes/latest/). * Review the release notes for plugins bundled with the latest version for any project-critical changes. Refer to the **Bundled plugin report** in the [CloudBees CD/RO release notes](/docs/cloudbees-cd-release-notes/latest/) of the latest version. Some of your plugins may have had several releases since your last integration. Refer the plugin’s release notes for a summary of changes. * Download the latest version’s Helm chart from the [CloudBees Nexus repo](https://public-charts.artifacts.cloudbees.com/#browse/browse:public) and compare its changes with your existing values file. Also note its `image.tag`, you will need this to upgrade the CloudBees CD/RO server image. * Create backups of your CloudBees CD/RO environment. For more information, refer to [Back up CloudBees CD/RO on Kubernetes](/docs/cloudbees-cd/latest/maintain-cdro/k8s-server-backups). ### [](#_upgrading_your_cloudbees_cdro_servers)Upgrading your CloudBees CD/RO servers To upgrade to the new CloudBees CD/RO version: 1. Make backups of your deployment and database. These backups may be necessary to recover data if critical errors occurs during the upgrade process. 2. Set the variables for the current release’s name, its namespace, and version you want to upgrade to: releaseName="" namespace="" # Example upgradeVersion value: "2.21.0" upgradeVersion="" 3. Save the values from your current release to a values file by running: helm get values $releaseName --namespace $namespace > old-values.yaml 4. Ensure your `old-values.yaml` contains values for the following parameters: * `ingress.host` * `storage.volumes.serverPlugins.storageClass` * `storage.volumes.repositoryStorage.storageClass` * `storage.volumes.analyticsStorage.storageClass` * `database.externalEndpoint` * `database.dbPort` * `database.dbType` * `database.dbName` * `database.dbUser` * `database.dbPassword` * `flowLicense` * `flowCredentials.adminPassword` If you have changed your admin password via the CloudBees CD/RO UI, it is critical you also update your Helm chart values or `existingSecret` with the new password prior to upgrading. Failing to do so, will cause an upgrade error: ectool error [AccessDenied]: User 'admin' must be local only. * `analytics.credentials.adminPassword` 5. If your `old-values.yaml` uses the `images.tag`, you must update it with the image tag of the new CloudBees CD/RO release you want to upgrade to. For more information, refer to [How does Kubernetes know which version to install](#_how_does_kubernetes_know_which_version_to_install). If you do not update the `images.tag` or update it with an incorrect value, your upgrade will not succeed. 6. For CloudBees CD/RO v2023.06.0 and later, this step is **OPTIONAL**. Scale down the `flow` deployments to `1` replica each by running: kubectl scale deployment flow-server -n $namespace --replicas=1 kubectl scale deployment flow-web -n $namespace --replicas=1 kubectl scale deployment flow-repository -n $namespace --replicas=1 7. Copy and save your `flow-server` passkey: # Get the flow-server pod name cdServerPod=$(kubectl get pod -l app=flow-server -o \ jsonpath='{.items[*].metadata.name}' -n $namespace); echo $cdServerPod # Get the passkey file name passkey=$(kubectl -n $namespace exec $cdServerPod -- ls /tmp/ | grep passkey); echo $passkey # Copy the passkey file from the flow-server pod to the local machine kubectl -n $namespace cp $cdServerPod:/tmp/$passkey ./passkey # Ensure the passkey file has been copied ls -l | grep passkey 8. Copy and store the `flow-server` keystore files: # Get the keystore file name keystore=$(kubectl -n $namespace exec $cdServerPod -- ls /tmp/ | grep keystore); echo $keystore # Copy the keystore file from flow-server pod to your local machine kubectl -n $namespace cp $cdServerPod:/tmp/$keystore ./keystore # Ensure the keystore file has been copied ls -l | grep keystore 9. To upgrade your release, run: helm search repo cloudbees helm pull cloudbees/cloudbees-flow --version $upgradeVersion tar -xf cloudbees-flow-$upgradeVersion.tgz helm dep update cloudbees-flow # If you used a name other than old-values.yaml, use it for --values helm upgrade $releaseName cloudbees-flow \ --values old-values.yaml \ --namespace $namespace \ # Helm upgrades exit when all tasks are complete, regardless of timeout. # However, if your database is very large, the timeout may need to be increased to prevent returning a FAILED status. # For instance: --timeout 10800s --timeout 4200s If you are installing a CloudBees CD/RO instance with an external database, do not use the `--wait` option. The `flow-server-init-job` cannot be started if `--wait` is used. The installation may take several minutes. Once the pods are all running, you can track the progress of the CloudBees CD/RO server initialization, via the `flow-server-init-job` log in the Kubernetes dashboard. If you run into issues during your upgrade, and need to restore CloudBees CD/RO, refer to [Restore to a previous CloudBees CD/RO Helm release](/docs/cloudbees-cd/latest/maintain-cdro/k8s-server-restore#restore-cdro-server-previous-version). [](#_bundled_plugins_updates)Bundled plugins updates ---------------------------------------------------- Starting with CloudBees CD/RO v2023.08.0, when upgrading your CloudBees CD/RO environment, bundled plugins are only updated or reinstalled if the bundled plugin version changes. This helps decrease upgrade times by skipping reinstalling bundled plugins that had no changes for the CloudBees CD/RO release. However, after a successful upgrade to v2023.08.0 or later, if you need to reinstall all bundled plugins regardless of whether they have changes, you can force an update to all bundled plugins. For CloudBees CD/RO v2023.06.0 or earlier, the following steps are not applicable. For each CloudBees CD/RO v2023.06.0 or earlier Kubernetes upgrade, all bundled plugins are reinstalled automatically. ### [](#_prerequisites_to_force_update_bundled_plugins)Prerequisites to force-update bundled plugins The following prerequisites are required to force all bundled plugins to reinstall: * You must successfully upgrade your current CloudBees CD/RO version to v2023.08.0 or later. * You must have CloudBees CD/RO administrator permissions and password for the `flow-server` instances where the bundled plugin updates should occur. ### [](#_force_update_bundled_plugins)Force-update bundled plugins After you have successfully upgraded your current CloudBees CD/RO version to v2023.08.0 or later, perform the following steps to force all bundled plugins to reinstall: 1. Find the pod name of the `flow-server` in your namespace: kubectl get pods --namespace -l app=flow-server 2. Use `kubectl exec` to access the `flow-server` pod and load a `bash` shell. kubectl exec --namespace -ti -c flow-server -- /bin/bash 3. Log into the `flow-server` instance as a CloudBees CD/RO administrator using `ectool`. ectool login admin 4. Provide your instance CloudBees CD/RO administrator password. 5. Execute the bundled plugin force-update using CloudBees `ectool`. ectool reloadSetupScripts --force 1 If no error message is returned, the command installs all bundled plugins, and will silently exit when complete. [](#_upgrade_cloudbees_cdro_agents_on_kubernetes)Upgrade CloudBees CD/RO agents on Kubernetes --------------------------------------------------------------------------------------------- For helpful planning considerations, refer to [Getting started](#_getting_started). If you only want to update your agent’s values without upgrading to a new CloudBees CD/RO version, refer to [CloudBees CD/RO agent installation and update](/docs/cloudbees-cd/latest/install-k8s/installation#Install-agent). Your CloudBees CD/RO server and agent(s) can have different CloudBees CD/RO versions and still operate together. However, once you have upgraded your CloudBees CD/RO to a new version, you can upgrade your agent(s) for version continuity. To upgrade your agent(s): 1. Make backups of your agent(s) deployment and any project-specific resources. These backups may be necessary to recover data if critical errors occurs during the upgrade process. 2. Set the variables for the current release’s name, its namespace, and version you want to upgrade to: releaseName="" namespace="" # Example upgradeVersion value: "2.21.0" upgradeVersion="" For `$namespace`, ensure you use the namespace of the `cloudbees-flow` instance where you want to upgrade the agent. Using the namespace of an incorrect instance will result in the agent being upgraded for that instance instead of your intended target. 3. Save the values from your current release to a values file by running: helm get values $releaseName --namespace $namespace > old-agent-values.yaml If you used the `--set-file customScript` option to install third-party tools with your agent, ensure they are included in the `old-agent-values.yaml` under the `customScript` tag. 4. Visit [CloudBees Nexus repo](https://public-charts.artifacts.cloudbees.com/#browse/browse:public), find the release of `cloudbees-flow-agent` for your upgrade, and download the package. 5. In the `cloudbees-flow-agent` package, locate and open the `values.yaml`. 6. In the `values.yaml`, find and copy the agent’s new release image from `images.tag`. 7. In your `old-values.yaml`, replace the `images.tag` with the agent’s new release image tag and save the file. 8. To upgrade your release, run: helm search repo cloudbees helm pull cloudbees/cloudbees-flow-agent --version $upgradeVersion tar -xf cloudbees-flow-agent-$upgradeVersion.tgz helm dep update cloudbees-flow-agent # If you used a name other than old-agent-values.yaml, use it for --values helm upgrade $releaseName cloudbees-flow-agent \ --values old-agent-values.yaml \ --namespace $namespace \ --timeout 4200s If you need to add or update third-party tools via script with your agent upgrade, use the following option: # Optional if you need to add or update third-party tools: --set-file customScript= For more information integrating third-party tools with your agent, refer to [Configure agent third-party tools](/docs/cloudbees-cd/latest/install-k8s/installation#agent-third-party-tools). The installation may take several minutes. Once the pods are all running, you can track the progress of the CloudBees CD/RO agent initialization via the Kubernetes dashboard.