Upgrading CloudBees CI on traditional platforms

Before upgrading

Take a SNAPSHOT of your CloudBees CI platform. For the operations center and every controller:

Generate a support bundle. It will help in case there are problems with the upgrade or to know what plugins were upgraded in the process.
Backup your instance before starting the upgrade. The minimal backup is a copy of your ${JENKINS_HOME} directory. If the BUILD_DIRECTORY is out of the ${JENKINS_HOME}, you must back it up as well.

Note: In the case something unexpected happens, the rollback process in based on the Backup of your instance.

For client controllers connected to operations center:

operations center must be upgraded first, then client controllers.
The client controller connected to the operations center does not have to be at the same version.

The version of operations center must always be more recent or as old as the version of the client controllers that are connected to the operations center (operations center >= client controller version).

Key principles to follow when upgrading from one version to the other

Change one thing at a time

Limit the changes you carry out to one area at a time. For example, do not plan to make infrastructure changes such as hardware upgrades, networking changes, OS package upgrades at the same time as upgrading Jenkins. This will help in identifying and isolating the root cause of any issues encountered.

Backup strategy

Ensure that you have regular backups of your JENKINS_HOME for your operations center, and all controllers. We recommend using the CloudBees Backup Plugin to automate this. controllers that have CloudBees Backup Plugin 3.38 or later installed have support for the Restore job type. To restore an operations center when using CloudBees CI on traditional platforms follow the steps for Restoring manually. If you are using CloudBees CI on modern cloud platforms, follow the steps to Backup and restore on Kubernetes. The backups should be stored at an offsite location such as cloud storage or at a minimum on separate hardware from your CloudBees CI.
Take another backup immediately before you start the upgrade.
Test that this backup is valid as we use it to recover from a failure if we encounter a severe issue during the production upgrade. A great way to test the validity of these backups is with a test environment. Another reason for testing the builds is that after an upgrade of the Jenkins core version or plugins, you cannot directly downgrade the war file and plugins, the only way to 'roll back' is to restore the JENKINS_HOME backup. To validate the integrity of the backup it is essential to extract the backup to another filesystem, and verify the checksums using:

  ----
  find $JENKINS_HOME -type f | sort | xargs md5sum >~/checksum.log.backup
  ----

Run this for both the live and backup data, and compare the checksum.log.* between the two.

Test strategy

We recommend you have a separate test environment with a similar setup to your production environment, with representative jobs. If you are creating a test environment and require a license please send us the instance ID of your test instance and we will send you a test instance license.
We recommend having jobs on your test controller that are representative of jobs on your production controller, so that testing the upgrade, uses the key plugins to your workflow, and the interactions with any external systems (SCM, artifact repositories, bug trackers, cloud automation). It is a best practice to build some jobs which are a good subset of representative jobs on this test cluster. These dedicated jobs represent your production jobs, but without working on your real data. We recommend talking to your development groups to find jobs that cover the main functionality. Here are some recommendations:
- test your end-to-end complex pipeline jobs
- test those jobs that utilize critical plugins
- run jobs which are part of integrations
- run jobs which are for specific technologies

To clone a production controller to a test controller, follow the steps from Migration Guide: CloudBees Jenkins Platform and CloudBees Jenkins Team , specifically, the section titled: "Case B - Migrate Entire Jenkins Configuration, Including Jobs" "Option A - Just Migration".

Update the configuration for some representative jobs so they do not collide with your production jobs, and run them to ensure they work correctly in the upgraded test instance. You will need to re-created credentials on your test instance.

This test environment can be useful after the upgrade as well, for example if you would like to test installing a new plugin.

Recommendations

Setup a test environment

This is ideally a full test environment with the operations center and client controllers and requires a test license. If this is not feasible, just having a test controller with representative jobs will work.

Enable CloudBees Assurance Program

Ensure that CloudBees Assurance Program is enabled to avoid plugin dependency issues or incorrect versions installed.

Upgrade to latest version

CloudBees recommends that you always upgrade to the latest version of your CloudBees product, ensuring a wider lifecycle support coverage plus more security patches added (CloudBees Security Advisory), more fixed issues, and more new features included.

Upgrade plugins

CloudBees recommends that you upgrade your plugins before upgrading a Jenkins instance. For instructions on upgrading plugins, see Upgrading plugins from the Plugin Manager. Run tests after upgrading the plugins to ensure that the upgrades are functioning as expected.

Upgrade instance where it is running

CloudBees highly recommends that you upgrade Jenkins to the same location where it is running even if it is in the production environment. Doing a backup of $JENKINS_HOME and the $BUILD_DIR - in case it is outside the default location - should be enough to revert to the previous status.

CloudBees does not recommend that you keep two different instances working at the same time. This is to avoid downtime while performing the upgrade unless you really know what you are doing - it will be very difficult to replicate the exactly same environment.

JNLP agent will not work correctly as the $JENKINS_URL location will be different.
Credentials might fail in case the secret is not the same on both instances.
The OS might not be configured in the same way. i.e ulimit.
The CloudBees CI configuration might not be correctly replicated.

Stop using Apache Maven builds (if possible)

The Maven plugin plugin is not recommended as it is considered unreliable by the Community. Therefore, it is recommended that you migrate jobs using this plugin to Pipeline Jobs with the Maven integration.

If (and only if) you are able to migrate your Maven plugin builds, please carefully read the Maven jobs and Java versions compatibility guide.

Upgrade process

Visit CloudBees downloads

Go to the downloads page. Select your desired distribution type, and review the instructions.

WAR
RPM
OpenSuse
Debian
Microsoft Windows
Docker

Installation procedures

WAR

If your CloudBees CI instance is run with the command java -jar cloudbees-core-cm.war or java -jar cloudbees-core-oc.war, you can simply replace the jenkins.war file with the latest version.

Linux

Red Hat / CentOS

Controller - yum upgrade cloudbees-core-cm
operations center - yum upgrade cloudbees-core-oc

RPM/YUM with High Availability

The RPM package contains a post-install script to ensure ownership on several files including JENKINS_HOME. Therefore if High Availability is setup, this could lead to I/O Errors - for example when upgrading one node while another one is still running. More information is available in JENKINS-23273.

To workaround this problem, you can skip this script by adding the property JENKINS_INSTALL_SKIP_CHOWN="true" under /etc/sysconfig/jenkins.

Debian / Ubuntu

Controller - apt-get update && apt-get install cloudbees-core-cm
operations center - apt-get update && apt-get install cloudbees-core-oc

openSUSE / SUSE Linux

Controller - zypper install cloudbees-core-cm
operations center - zypper install cloudbees-core-oc

Servlet container

Once a .war archive has been downloaded, follow the servlet container’s existing application deployment process.

When using servlet containers, CloudBees CI will set the JENKINS_HOME to the $APP_SERVER_USER/.jenkins/ folder. If the servlet container installation does not include write permissions to this folder for this user (sometimes done for security), you either need to grant appropriate permissions or override this setting by adding the "-DJENKINS_HOME=$MY_JENKINSPATH" argument in your servlet container startup. Refer to the servlet container’s documentation for how to add startup arguments.

Tomcat

Use environment variable CATALINA_OPTS to add:

-Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true.
-Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true which is needed for Blue Ocean.

CloudBees recommends that you configure it in the script $CATALINA_BASE/bin/setenv.sh (Linux) or %CATALINA_BASE%\bin\setenv.bat (Microsoft Windows) that you’ll create to customize your application server.

Custom container installations

If you use a custom container, you will find the cloudbees-core-cm.war or cloudbees-core-oc.war file in the deploy directory of your container. For example, /usr/local/jboss/server/default/deploy/cloudbees-core-cm.war would be the location for a default JBoss installation.

Microsoft Windows

To upgrade on Microsoft Windows servers, a new .zip must be downloaded from the downloads page and installed.

Upgrade steps

Before you upgrade

Ensure that you have enabled the CloudBees Assurance Program (CAP), so that when you go to upgrade the plugins during the upgrade process, you will be upgrading to versions tested by CAP. For more information, please see Upgrading plugins with Beekeeper Upgrade Assistant.
Enable the CloudBees Quiet Start plugin so that when the instance first comes online after the upgrade, builds will not start automatically before you get a chance to upgrade the plugins.
Ensure that the test environment is functioning correctly after it was upgraded.
Ensure that you have taken a backup.
Ensure that all running builds complete.
Java 8 is now required for everything on your Jenkins instance, operations center, Jenkins controllers and all agents starting with Jenkins LTS 2.60.1. For more information, please see link: Upgrade guide here.

Recommended steps

Upgrade the plugins on the operations center. To upgrade:

Go to Manage Jenkins → Beekeeper Upgrade Assistant → CAP Configuration.
Ensure that the Enroll this instance in the CloudBees Assurance Program checkbox is checked.
Ensure that the Allow automatic upgrades of plugins on restart checkbox is checked.
Click Save.
Go to Manage Jenkins → Manage Plugins and under the Updates tab, click upgrade all plugins.

Restart the operations center.

If you are using the cas-plugin, and are upgrading past version 2.277.1.2, please do not upgrade it using the Plugin Manager, it must be upgraded at the same time as the instance. This means that once the service is stopped, you will need to link:http://jenkins-updates.cloudbees.com/download/plugins/cas-plugin/ [download the plugin] and manually replace $JENKINS_HOME/plugins/cas-plugin.jpi.

Upgrade the Kubernetes resources and the CloudBees Jenkins operations center.
- To upgrade using helm, follow Upgrading CloudBees CI on modern cloud platforms.
- If you are not already using Helm, we recommend to migrate to it as per Existing CloudBees CI installations to Helm migration guide.
The next step is to upgrade the agent.jar file for inbound agents that do not use service wrapper’s auto-update. To update the file, make sure that that you have done step 5 in the recommended steps section and follow the instructions in the Upgrading CloudBees Jenkins Platform - Post-upgrade notes section of the CJP upgrade guide.

The only section of that upgrade guide that is relevant is the Post-upgrade notes section.
Upgrade the managed controllers. For instructions please see Managing Masters - Upgrading managed controllers. Please follow the same plugin upgrade steps you followed for {OS}. A full backup should be taken before you upgrade each controller. Go to Manage Jenkins → Prepare for Shutdown and wait for any running builds to complete before you upgrade a controller.
Now that you are running the latest version of CloudBees CI, we recommend you enable actionable build notifications in GitHub and Bitbucket and slack. For more details, see Enabling actionable build notifications in GitHub and Bitbucket and Setting up actionable build notifications in Slack. This generally takes just a few minutes and provides enhanced notifications for your developers with no pipeline changes required.
When updating your controllers we recommend you also install the CloudBees Jenkins Health Advisor plugin if not already installed. For instructions please see , Jenkins Health Advisor by CloudBees. CloudBees Jenkins health Advisor automatically identifies issues that could impact the performance, stability, and security of your controller and it also identifies potential issues due to known issues (such as plugin bugs) on your controller. You get notified via email when a new problem is found, and the email includes links to the solutions for those identified issues. We recommend all clients enable it.

If you are using the High Availability (HA) feature, the general process for updating the HA operations center or any HA controller is:

Stop both HA nodes
Upgrade both HA nodes
Start one of the HA nodes, wait for it to come fully online
Update the plugins according to CAP recommendations and restart
Start the other HA node

Post-upgrade notes

In case, JNLP agents (specifically for Microsoft Windows) are not able to connect, review the article Windows agent offline or unable to connect.

Review articles in our Best Practices and apply changes accordingly.