IntroductionPipeline termsPipeline project typesCloudBees Pipeline benefits
IntroductionUsing Pipeline development utilitiesDetermining plugin compatibilityPipeline best practices
IntroductionPrerequisitesCreating your first PipelineCreating Pipelines in SCMCreating Pipelines in Web UIUsing Pipeline as Code
IntroductionConfig Adv ScriptedConfig Build StageConfig Deploy StageConfig Optional Step ArgsConfig Test StageCreating JenkinsfileCustomizing ParametersHandling FailuresString InterpolationUsing Multiple AgentsWorking With The Env
IntroductionManaging artifactsTracing artifactsTriggering jobs with a simple webhookRestoring filesVisualizing PipelineInserting checkpointsSecuring PipelinesConfiguring Pipelines with user-scoped credentialsUsing Pipeline PoliciesSpecifying a matrix of one or more dimensionsConverting a Freestyle project to a Declarative Pipeline
IntroductionRestarting aborted buildsLong-running buildsSkip next buildConsolidated Build View pluginCloudBees Quiet Start pluginCloudBees Template pluginCloudBees Git Validated Merge plugin
IntroductionTrigger a job with a notification event using Cross Team CollaborationEnable external notification events with external HTTP endpointsCluster-wide copy artifactsCluster-wide job triggers
IntroductionSetting up a Pipeline Template CatalogDefining Pipeline Template CatalogsSetting Up A Pipeline TemplateUsing parameters in template.yamlManaging multibranch Pipeline options in template.yamlManaging Pipeline Template Catalogs in bulkExample full Maven/Java app Jenkinsfile
IntroductionUsing Declarative Pipeline syntaxUsing Scripted Pipeline syntax
IntroductionBranch SourceBitbucketGitBranch Property StrategyExamples
CloudBees Docker Build and Publish pluginCloudBees Docker TraceabilityDocker Pipeline pluginCloudBees Docker Hub/Registry Notification plugin
IntroductionExplaining $JENKINS_HOMEBest practices for backup and restoreBacking up manuallyRestoring manuallyRestoring credentialsUsing the CloudBees Backup PluginScheduling your backups in the CloudBees Backup PluginRestoring from CloudBees Backup PluginUsing cluster operations to run backups
IntroductionGetting the CLI toolUsing the CLI toolExamples of usageConfiguring multiple client controllersConfiguring an alias for the Jenkins CLI toolConfiguring the Jenkins CLI tool to work with non-TrustStore SSL certificatesCasC bundle management on operations center
CloudBees Inactive Items Plugin
Jenkins Health Advisor by CloudBees
CloudBees Pull Request Builder for GitHub plugin
Counting and monitoring user licenses with the CloudBees User Activity Monitoring plugin
Support policies

Upgrading CloudBees Jenkins Platform

7 minute read

Before upgrading

Take a SNAPSHOT of your CloudBees Jenkins Platform. For the operations center and every controller:

  • Generate a support bundle. It will help in case there are problems with the upgrade or to know what plugins were upgraded in the process.

  • Backup your instance before starting the upgrade. The minimal backup is a copy of your ${JENKINS_HOME} directory. If the BUILD_DIRECTORY is out of the ${JENKINS_HOME}, you must back it up as well.

Note: In the case something unexpected happens, the rollback process is based on the Backup of your instance.

For client controllers connected to the operations center:

  • The operations center must be upgraded first, and then client controllers.

  • Starting with CloudBees Jenkins Platform 2.7 and for all versions of CloudBees CI on traditional platforms, the version of operations center must always be more recent or as old as the version of the client controllers that are connected to the operations center (operations center >= client controller version).

  • The client controller connected to a operations center does not have to be at the same version.

Key principles to follow when upgrading

Change one thing at a time

Limit the changes you make to one area at a time. For example, do not plan to make infrastructure changes such as hardware upgrades, networking changes, or OS package upgrades when upgrading Jenkins. This helps to identify and isolate the root cause of any issues you encounter.

Backup strategies

CloudBees recommends the following backup strategies:

  1. Ensure you have regular backups of your JENKINS_HOME for the operations center and all controllers.

    CloudBees recommends using the CloudBees Backup Plugin plugin to automate this. controllers that have the CloudBees Backup Plugin 3.38 or later installed have support for the Restore job type.

    • To restore an operations center when using CloudBees Jenkins Platform, follow the steps for Restoring manually.

      The backups should be stored at an offsite location such as cloud storage or, at a minimum, on separate hardware from your CloudBees CI.

  2. Take another backup immediately before you start the upgrade.

  3. Test the backup to verify it is valid.

    The backup is used to recover from a failure if you encounter a severe issue during the production upgrade.

    After an upgrade of the Jenkins core version or plugins, you cannot directly downgrade the war file and plugins. The only way to roll back is to restore the Jenkins_HOME backup.

    To validate the integrity of the backup, it is essential to extract the backup to another filesystem and verify the checksums using:

    find $JENKINS_HOME -type f | sort | xargs md5sum >~/checksum.log.backup
    ▼

    Run this for both the live and backup data, and compare the checksum.log.* between the two.

Test strategies

CloudBees recommends the following test strategies:

  1. Create a separate test environment that has a similar setup as your production environment with representative jobs.

    If you are creating a test environment and require a license, please send CloudBees the instance ID for your test instance, and CloudBees will send you a test instance license.

  2. Create jobs on your test controller that are representative of jobs on your production controller. This allows you to test the upgrade using the key plugins for your workflow and interactions with any external systems such as SCM tools, artifact repositories, bug trackers, and cloud automation.

    It is a best practice to create jobs that are a subset of representative jobs on the test cluster. These dedicated jobs represent your production jobs, but without working on real data. CloudBees recommends talking to your development groups to find jobs that cover the primary functionality, such as:

    • Test your end-to-end complex pipeline jobs.

    • Test jobs that utilize critical plugins.

    • Run jobs that are part of integrations.

    • Run jobs that are for specific technologies.

      To clone a production controller to a test controller, follow the steps from Migration Guide: CloudBees CI, specifically the section titled, Case B - Migrate Entire Jenkins Configuration, Including Jobs.

  3. Update the test configuration for some representative jobs so they do not collide with your production jobs. Then, run the test jobs to ensure they work correctly in the upgraded test instance. You will need to recreate credentials on your test instance.

This test environment can also be useful after the upgrade to test the installation of new plugins.

Recommendations

Enable CloudBees Assurance Program

Ensure that CloudBees Assurance Program is enabled to avoid plugin dependency issues or incorrect versions installed.

Upgrade to latest version

It is recommended always to upgrade to the latest version of your CloudBees product, ensuring a wider lifecycle support coverage plus more security patches added (CloudBees Security Advisory), more fixed issues, and more new features included.

JENKINS_HOME should be in the same location

It is highly recommended to upgrade Jenkins to the same location where it is running even if it is in the production environment. Doing a backup of $JENKINS_HOME and the $BUILD_DIR - in case it is outside the default location - should be enough to revert to the previous status.

It is not recommended to keep two different instances working at the same time to avoid downtime while performing the upgrade unless you really know what you are doing - it will be very difficult to replicate the exactly same environment.

  • JNLP agent will not work correctly as the $JENKINS_URL location will be different.

  • Credentials might fail in case the secret is not the same on both instances.

  • The OS might not be configured in the same way. i.e ulimit.

  • The Jenkins configuration might not be correctly replicated.

Stop using Apache Maven builds (if it is possible)

Maven plugin plugin is not recommended as it is often considered evil by the Community. Therefore, it is recommended that you migrate jobs using this plugin to Pipeline Jobs with the Maven integration.

If (and only if) you are able to migrate your Maven plugin builds, please read carefully the Maven jobs and Java versions compatibility guide.

Upgrade process

Visit CloudBees downloads

Go to the downloads page. Select your desired distribution type, and review the instructions.

  • WAR

  • RPM

  • OpenSuse

  • Debian

  • Microsoft Windows

  • Docker

Installation procedures

WAR

If your CloudBees Jenkins Platform instance is run with the command java -jar jenkins.war or java -jar jenkins-oc.war, you can simply replace the jenkins.war/jenkins-oc.war file with the latest version.

Linux

Red Hat / CentOS

Beginning in CloudBees Jenkins Platform version 2.319.1.5, the epel-release package is required to upgrade CloudBees Jenkins Platform using an RPM, however some CentOS distributions do not include it. For those CentOS distributions, run the following command before you begin the installation procedure: sudo yum install epel-release.

  • Controller - yum upgrade jenkins

  • Operations center - yum upgrade jenkins-oc

IMPORTANT: RPM/YUM with High Availability

The RPM package contains a post-install script to ensure ownership on several files including JENKINS_HOME. Therefore if High Availability is setup, this could lead to I/O Errors - for example when upgrading one node while another one is still running. More information is available in JENKINS-23273.

To workaround this problem, you can skip this script by adding the property JENKINS_INSTALL_SKIP_CHOWN="true" under /etc/sysconfig/jenkins.

Debian / Ubuntu

  • Controller - apt-get update && apt-get install jenkins

  • Operations center - apt-get update && apt-get install jenkins-oc

openSUSE / SUSE Linux

  • Controller - zypper install cloudbees-core-cm

  • Operations center - zypper install cloudbees-core-oc

Servlet container

Once a .war archive has been downloaded, follow the servlet container’s existing application deployment process.

When using servlet containers, CloudBees Jenkins Enterprise will set the JENKINS_HOME to the $APP_SERVER_USER/.jenkins/ folder. If the servlet container installation does not include write permissions to this folder for this user (sometimes done for security), you either need to grant appropriate permissions or override this setting by adding the "-DJENKINS_HOME=$MY_JENKINSPATH" argument in your servlet container startup. Refer to the servlet container’s documentation for how to add startup arguments.

Tomcat

Use environment variable CATALINA_OPTS to add:

  • -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true.

  • -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true which is needed for Blue Ocean.

It is recommended to configure it in the script $CATALINA_BASE/bin/setenv.sh (Linux) or %CATALINA_BASE%\bin\setenv.bat (Microsoft Windows) that you’ll create to customize your application server.

Custom container installations

If you use a custom container, you will find the jenkins.war or jenkins-oc.war file in the deploy directory of your container. For example, /usr/local/jboss/server/default/deploy/jenkins.war would be the location for a default JBoss installation.

Microsoft Windows

To upgrade on Microsoft Windows servers, a new .zip must be downloaded from the downloads page and installed.

Upgrade steps

Before you upgrade

  1. Ensure you have enabled the CloudBees Assurance Program (CAP) to ensure you are upgrading to plugin versions tested by CAP. For more information, please refer to Upgrading plugins with Beekeeper Upgrade Assistant.

  2. Enable the CloudBees Quiet Start Plugin plugin to ensure that when the instance first comes online after the upgrade, builds will not start automatically before you get a chance to upgrade the plugins.

  3. Verify the test environment is functioning correctly after it is upgraded.

  4. Ensure you have taken a backup.

  5. Verify all running builds are complete.

  6. Java 11 is recommended for your Jenkins instance, the operations center, Jenkins controllers, and all agents for Jenkins LTS 2.303.3. For more information, refer to the upgrade guide.

CloudBees strongly recommends that you upgrade to Java 11 as soon as possible. The ability to run new versions of the product on Java 8 will be removed in an upcoming release. Releases that occurred during the support window for Java 8 will continue to run on Java 8.

Recommended steps

  1. Upgrade the plugins on the operations center.

    1. Select Manage Jenkins  Beekeeper Upgrade Assistant  CAP Configuration.

    2. Verify that Enroll this instance in the CloudBees Assurance Program is selected.

    3. Verify that Allow automatic upgrades of plugins on restart is selected.

    4. Select Save.

    5. Navigate to Manage Jenkins  Manage Plugins and under the Updates tab, select upgrade all plugins.

    6. Restart the operations center.

      If you are using the CAS plugin and upgrading to version 2.277.1.2 or later, do not upgrade it using the Plugin Manager. It must be upgraded at the same time as the instance. Once the service is stopped, you must download the CAS plugin and manually replace $JENKINS_HOME/plugins/cas-plugin.jpi.

  2. Upgrade the operations center and controllers.

  3. After your controllers are upgraded, you must manually upgrade the agent.jar file for any inbound agents that are not configured to upgrade the agent.jar file automatically. For more information, refer to Windows agent offline or unable to connect.

  4. Now that you are running the latest version of CloudBees CI, CloudBees recommends you enable actionable build notifications in GitHub, Bitbucket, and Slack. This generally takes a few minutes and provides enhanced notifications for your developers with no Pipeline changes required. For more information, refer to Enabling actionable build notifications in GitHub and Bitbucket and Setting up actionable build notifications in Slack.

  5. When updating your controllers, CloudBees recommends that you install the Health Advisor by CloudBees plugin, if not already installed. The CloudBees Jenkins Health Advisor automatically identifies issues that could impact the performance, stability, and security of your controller. It also identifies potential issues due to known issues on your controller. You are notified via email when a new problem is found and the email includes links to the solutions for the identified issues. For instructions, refer to Jenkins Health Advisor by CloudBees.

    If you are using the High Availability (HA) feature, the general process for updating the HA operations center or any HA controller is:

    1. Stop both HA nodes.

    2. Upgrade both HA nodes.

    3. Start one of the HA nodes and wait for it to come fully online.

    4. Update the plugins according to CAP recommendations and restart.

    5. Start the other HA node.

Post-upgrade notes

Review articles in our Best Practices and apply changes accordingly.

Migrating to Java 11