Introduction
IntroductionPre-installation requirementsInstallVerify Docker imagesUninstall
IntroductionPre-installation requirementsInstallVerify Docker imagesUninstall
IntroductionPre-installation requirementsInstallVerify Docker imagesUninstall
IntroductionPre-installation requirementsInstallVerify Docker imagesUninstall
IntroductionPre-installation requirementsInstallVerify Docker imagesUninstall
IntroductionPre-installation requirementsInstallVerify Docker imagesUninstall
IntroductionSystem requirementsVerify Docker imagesInstall operations centerInstall client controllersVerify build componentsVerify WAR files
IntroductionHA fundamentalsGet ready for HAInstall HA on modern cloud platformsInstall HA on traditional platformsHA considerationsHigh Availability (active/passive) installation for CloudBees CI on traditional platforms
IntroductionWhat is FIPS and FIPS 140 compliance?Install CloudBees CI on modern cloud platforms in FIPS modeCAP plugin support in a FIPS 140-3 environmentConfigure the Apache™ Ant plugin for FIPS complianceConfigure the Pipeline Maven API plugin for FIPS complianceKnown FIPS incompatibilities with CloudBees CI on modern cloud platformsJenkins core: FIPS 140-3 compliant artifacts with caveatsJenkins core: Non-compliant classes and libraries
Introduction
IntroductionCommon Pipeline termsPipeline project typesCloudBees proprietary features for Pipelines
IntroductionPipeline development utilitiesDetermine plugin compatibilityPipeline best practices
IntroductionPipeline prerequisitesCreate your first PipelineSelect an agent for your Pipeline jobCreate a Pipeline from SCMCreate a Pipeline in the UIUnderstand and implement Pipeline as Code
IntroductionConfigure advanced Scripted PipelineConfigure the build stageConfigure the deploy stageConfigure optional step argumentsConfigure the test stageCreate a JenkinsfileCustomize parametersHandle failuresString interpolationUse multiple agentsWork with the environmentReuse configuration files
IntroductionManage artifacts with CloudBees Fast Archiving pluginEnable artifact traceability with fingerprintingTrigger jobs with a simple webhookRestore filesVisualize the PipelineInsert checkpointsSecure PipelinesConfigure Pipelines with user-scoped credentialsEnforce standards with Pipeline PoliciesSpecify a matrix of one or more dimensionsConvert a Freestyle project to a Declarative PipelinePipeline builds and High Availability (active/active)
IntroductionRestart aborted buildsLong-running buildsSkip next buildConsolidated Build View pluginCloudBees Quiet Start pluginCloudBees Template pluginCloudBees Workspace Caching
IntroductionTrigger a job with a notification event using Cross Team CollaborationEnable external notification events with external HTTP endpointsCluster-wide copy artifactsCluster-wide job triggers
IntroductionSet up a Pipeline Template CatalogDefine Pipeline Template CatalogsSet up a Pipeline TemplateParameter types in the template.yaml fileManage Multibranch Pipeline options in the template.yaml fileManage Pipeline Template Catalogs in bulkExample full Maven/Java app Jenkinsfile
IntroductionUse Declarative Pipeline syntaxUse Scripted Pipeline syntax
IntroductionBranch SourceBitbucketGitHubGitBranch Property StrategyExamples
CloudBees Docker Build and Publish pluginCloudBees Docker TraceabilityDocker Pipeline pluginCloudBees Docker Hub/Registry Notification plugin
Introduction
Introduction
IntroductionGet startedNavigate the operations center interfaceProvision agents in a separate Kubernetes cluster from a managed controllerProvision a controller in a different OpenShift project than the operations centerManage controllersManage controllers in specific Kubernetes namespacesMigrate an existing managed controller to High Availability (HA)Manage agentsManage SSH credentialsShared agentsShared configurationsShared cloud configurationTrigger restrictionsChange NFS storage locationQuiet startMove/Copy/PromoteCluster operationsInbound agentsUse Kaniko with CloudBees CIUse Buildkit with CloudBees CIUsing self-signed certificates in CloudBees CI on KubernetesSidecar injector for self-signed certificates on OpenShiftAuto-scale nodes on EKSEnable auto-scaling nodes on GKECloudBees Amazon Web Services Deploy EngineCloudBees Amazon AWS CLI pluginCloud Foundry CLI PluginIntegrate OpenShift CLICloudBees CI ServiceNow integrationCreate projects based on a GitHub repository structureUse GitHub App authenticationCreate Multibranch Projects and Organization Folders with large repositoriesWikiText pluginController Lifecycle Notifications Plugin
IntroductionGet startedNavigate the operations center interfaceManage client controllersMigrate from High Availability (active/passive) to High Availability (active/active) on CloudBees CI on traditional platformsManaging agentsManage SSH credentialsShared agentsShared configurationsShared cloud configurationTrigger restrictionsQuiet startMove/Copy/PromoteCluster operationsInbound agentsCloudBees CI ServiceNow integrationCreate projects based on a GitHub repository structureUse GitHub App authenticationCreate Multibranch Projects and Organization Folders with large repositoriesWikiText plugin
Introduction
Trust model
Pod Security Admission
Centrally manage security for controllers
CyberArk Credential Provider Plugin
CloudBees HashiCorp Vault Plugin
Enable advanced use cases: Cross-controller triggers and bulk operations
Restrict access and delegate administration with Role-Based Access Control
Example RBAC configurations
Create secure folders with Role-Based Access Control Auto Configurer
Restrict job triggers
Set up operations center-level access controls
Set up access controls on connected controllers
Test the SSH connection to an agent
Configure restricted credentials with the CloudBees Restricted Credentials plugin
Folders plugin
Folders Plus plugin
Inject secrets into builds
Manage build agents with the Nodes Plus plugin
Extended security settings
Enhanced credentials masking
Understand Beekeeper security warnings
Prevent unauthorized interaction between builds
Security recommendations
Use single sign-on (SSO) in the operations center
CloudBees CI integration with Microsoft Entra ID
Operations center specific permissions
Authentication mapping
Delegate administration
Data collection for the CloudBees Analytics Plugin
Modern DevOps external secrets management
Replace an expired certificate
List of URLs that need access
Block access to URL patterns using the CloudBees Request Filter Plugin
Serve resources from Jenkins
Verify Helm charts with a signature
Introduction
Introduction$JENKINS_HOME directoryBest practices for backup and restoreBack up $JENKINS_HOME manuallyBackup a Role-Based Access Control configurationRestore backup files manuallyRestore credentialsConfigure backups using the CloudBees Backup pluginSchedule backups in the CloudBees Backup pluginRestore backups created with the CloudBees Backup pluginBackup and restore on KubernetesBackup and restore on AWSBackup and restore Kubernetes cluster resources using VeleroRun backups using cluster operations
IntroductionConfigure multiple client controllers with the Jenkins CLI toolConfigure an alias for the Jenkins CLI toolConfigure Jenkins CLI tool with non-TrustStore TLS certificates
CloudBees Inactive Items Plugin
Jenkins Health Advisor by CloudBees
CloudBees Pull Request Builder for GitHub plugin
Count and monitor user licenses with the CloudBees User Activity Monitoring plugin
CloudBees CI Catalog Plugin - Catalog tool
Introduction
CasC fundamentalsCasC requirementsCasC permissionsRecommended workflow
IntroductionExport a CasC configurationTransform an exported bundleAdvanced CasC bundle configurationValidate a CasC bundle
CloudBees CI CasC for operations centersGet started with Configuration as Code for the operations centerConfigure the operations center on modern platforms using CasCConfigure the operations center on traditional platforms using CasCRetrieve bundles using an SCMTroubleshoot CasC
CloudBees CI CasC for controllersGet started with Configuration as Code for controllersDistribute CasC bundles to controllers from your operations centerAdd controller CasC bundles to the operations centerConfigure bundle availability for controllersSet up a client controller using CasCSet up a managed controller using CasCSet up a managed controller using the CasC Controller Bundle ServiceAdvanced topicsTroubleshoot CasC for controllers
IntroductionUpdate a CasC bundleBundle update timingReview the CasC update log
IntroductionPlugin management with CasCDetermine plugin compatibility using CasCCreate an alternate plugin download site
Create items using CasC
Configure RBAC with CasC
CasC CLI commands
CasC HTTP API
Introductionbundle.yaml file referencejenkins.yaml file referenceplugins.yaml file referenceplugin-catalog.yaml file referenceitems.yaml file referencerbac.yaml file referencevariables.yaml file reference
HA on EKS Performance Test
Support policies

Upgrade CloudBees CI on traditional platforms

9 minute read

This guide describes the upgrade process for CloudBees CI on traditional platforms. CloudBees recommends that you always upgrade to the latest version of your CloudBees product, ensuring wider lifecycle support coverage, plus more security patches added via the CloudBees Security Advisory, resolved issues, and new features.

  • For client controllers that are connected to the operations center:

    • The operations center must be upgraded first, and then client controllers can be upgraded.

    • The client controller connected to the operations center does not have to be at the same version as the operations center. The operations center version must always be more recent or as old as the version of the client controllers that are connected to the operations center (operations center >= client controller version). CloudBees supports upgrading CloudBees CI to a version up to one year later than your current version.

      For more information, refer to Version skew between the operations center and controllers.

  • Java 21 is recommended for your CloudBees CI instance. The Jenkins community supports running Jenkins with Java 21 since version 2.426.1 and CloudBees CI supports running in Java 21 since version 2.479.3.1.

CloudBees strongly recommends that you upgrade to Java 21 as soon as possible. The ability to run new versions of the product on Java 17 will be removed in an upcoming release. Releases that occurred during the support window for Java 17 will continue to be Java 17 compatible. For information on how to migrate to Java 17 or Java 21, refer to:

Please contact CloudBees Support if you have any concerns or questions.

As a best practice, CloudBees recommends that you upgrade the operations center first. Then, you should upgrade any agents and controllers at the same time. If you use any customized pod templates in your environment, make sure they are also running the same Java version.

Choose an upgrade target

CloudBees supports upgrading your CloudBees CI instance to a version up to one year later than your current version, as documented in Upgrade between versions.

For this reason, upgrade your CloudBees CI environment on a regular basis. If your instance is more than a year old, multiple upgrades will be required to upgrade to the latest release.

If running versions older than 2.452.2.3, follow the specific version guidance from Upgrading from CloudBees CI versions older than 2.452.2.3

Prepare for the upgrade

Before upgrading, CloudBees recommends that you complete the following steps to prepare for the upgrade.

  1. Review the best practices for upgrading.

  2. Generate a support bundle.

  3. Enable the CloudBees Assurance Program.

  4. Back up your instance.

  5. Create a test environment.

  6. [Install the Health Advisor by CloudBees plugin to your controllers].

  7. [Install the CloudBees Quiet Start plugin to your controllers].

Best practices when upgrading

When upgrading, CloudBees highly recommends the following best practices.

  • Change one thing at a time: Limit the changes you make to one area at a time. For example, do not plan to make infrastructure changes such as hardware upgrades, networking changes, or OS package upgrades when upgrading. This helps to identify and isolate the root cause of any issues you encounter.

  • Upgrade the instance where it is running: Upgrade your instance in the same location where it is currently running, even for production environments.

    CloudBees does not recommend that you keep two different production instances running at the same time. This may be done to avoid downtime while performing the upgrade. However, unless you really know what you are doing, it is very difficult to replicate the exact same environment. For example:

    • The inbound agent does not work correctly as the $JENKINS_URL because the location is different.

    • Credentials may fail if the secret is not the same for both instances.

    • The OS may not be configured in the same way. For example, ulimit.

    • The CloudBees CI configuration may not be correctly replicated.

  • Upgrade the CAS plugin at the same time as the instance: If you are using the CAS plugin and upgrading to version 2.277.1.2 or later, do not upgrade it using the Plugin Manager. It must be upgraded at the same time as the instance. Once the service is stopped, you must download the CAS plugin and manually replace $JENKINS_HOME/plugins/cas-plugin.jpi.

  • Stop using the Maven plugin: The Maven Integration plugin is not recommended because it is considered unreliable by the Jenkins Community. CloudBees recommends that you migrate jobs using the Maven Integration plugin to Pipeline Jobs using the Pipeline Maven Integration plugin. If you are able to migrate your jobs, please carefully read the Maven jobs and Java versions compatibility guide.

    End-of-life announcement

    CloudBees has determined that this plugin will be removed from the CloudBees Assurance Program (CAP) in late 2022. Notification will be provided prior to the removal of the plugin. Please contact CloudBees Support if you have any concerns or questions.

Generate a support bundle

Before upgrading, CloudBees recommends generate a support bundle. It can help you troubleshoot problems with the upgrade and determine which plugins were upgraded in the process.

Enable the CloudBees Assurance Program

Before upgrading, CloudBees recommends that you enable the CloudBees Assurance Program (CAP) to ensure you are upgrading to plugin versions that are tested by CAP, to avoid plugin dependency issues and prevent incorrect versions from being installed.

To enable CAP and upgrade your plugins:

  1. Navigate to Manage Jenkins  Beekeeper Upgrade Assistant  CAP Configuration.

  2. Verify that Enroll this instance in the CloudBees Assurance Program is selected.

  3. Verify that Allow automatic upgrades of plugins on restart is selected.

  4. Select Save.

Back up your instance

Before upgrading, CloudBees recommends taking regular backups of your $JENKINS_HOME directory for the operations center and all controllers. The backups should be stored at an offsite location such as cloud storage or, at a minimum, on separate hardware from CloudBees CI.

To back up your instance:

  1. Back up the operations center and all controllers. CloudBees recommends using the CloudBees Backup plugin to automate the backup process. For more information, refer to Configure backups using the CloudBees Backup plugin.

    • You can also perform a minimal backup by making a copy of your $JENKINS_HOME directory for the operations center and all controllers. If the $BUILD_DIRECTORY is not located in the default location within the $JENKINS_HOME directory, you must back it up as well. If something unexpected occurs, the rollback process is based on the backup of your instance. For more information, refer to Back up $JENKINS_HOME manually.

    • Controllers that have the CloudBees Backup plugin 3.38 or later installed support the Restore job type. The operations center does not have a restore option, so to restore the operations center when using CloudBees CI, you can follow the steps for Restore backup files manually.

  2. Test the backup to verify it is valid. The backup is used to recover from a failure if you encounter a severe issue during the production upgrade.

    After an upgrade of the Jenkins core version or plugins, you cannot directly downgrade the war file and plugins. The only way to roll back is to restore the $JENKINS_HOME backup.

  3. Validate the integrity of the backup by extracting the backup to another filesystem and verifying the checksums using:

    find $JENKINS_HOME -type f | sort | xargs md5sum >~/checksum.log.backup
    ▼

    Run this for both the live and backup data, and compare the checksum.log.* between the two.

Create a test environment

Before upgrading, CloudBees recommends creating a separate test environment that has a similar setup as your production environment with representative jobs. This is ideally a full test environment with the operations center and client controllers and requires a test license. If this is not possible, using a test controller with representative jobs is enough.

If you are creating a test environment and require a license, please send the instance ID for your test instance to CloudBees Support, and CloudBees will send you a test instance license. While you are waiting for the permanent license, you can use the Request a trial license button in the CloudBees CI UI, as long as your instance has internet access and can connect to the CloudBees License Registry.

Create jobs on your test controller

Create jobs on your test controller that are representative of jobs on your production controller. This allows you to test the upgrade using the key plugins for your workflow and interactions with any external systems such as SCM tools, artifact repositories, bug trackers, and cloud automation. This test environment can also be useful after the upgrade to test the installation of new plugins.

It is a best practice to create jobs that are a subset of representative jobs on the test cluster. These dedicated jobs represent your production jobs, but without using real data. CloudBees recommends talking to your development groups to find jobs that cover the primary functionality, such as:

  • Testing your end-to-end complex Pipeline jobs.

  • Testing jobs that utilize critical plugins.

  • Running jobs that are part of integrations.

  • Running jobs that are for specific technologies.

    To clone a production controller to a test controller, follow the steps from Migration Guide: CloudBees CI, specifically the section titled, Case B - Migrate Entire Jenkins Configuration, Including Jobs.

Run the test jobs

Once you have created the jobs, you can run them on your test controller.

To run test jobs:

  1. Update the test configuration for some representative jobs, so they do not collide with your production jobs.

  2. Run the test jobs to ensure they work correctly in the upgraded test instance. You must recreate credentials on your test instance.

Upgrade your plugins in the test environment

CloudBees recommends that you upgrade the plugins in your test environment. After upgrading the plugins, you can run tests to ensure the upgraded plugins are functioning as expected.

Install the Health Advisor by CloudBees plugin to controllers

Before upgrading your controllers, CloudBees recommends installing the Health Advisor by CloudBees plugin, if it is not already installed. The Health Advisor by CloudBees automatically identifies issues that could impact the performance, stability, and security of your controllers. It also identifies potential issues due to known issues on your controllers. You are notified via email when a new problem is found and the email includes links to the solutions for the identified issues. For instructions, refer to Health Advisor by CloudBees plugin.

Install the CloudBees Quiet Start plugin to controllers

Before upgrading, CloudBees recommends installing the CloudBees Quiet Start plugin, if it is not already installed. The Quiet Start plugin ensures that builds do not start automatically when the instance first comes online after the upgrade.

Upgrade the operations center and controllers

Once you have upgraded your test environment and verified it is functioning as expected, you can perform the upgrade in your production environment.

If you are using the High Availability (HA) feature, the general process for upgrading the HA operations center or any HA controller is:

  1. Stop both HA nodes.

  2. Upgrade both HA nodes.

  3. Start one of the HA nodes and wait for it to come fully online.

  4. Upgrade the plugins according to CAP recommendations and restart.

  5. Start the other HA node.

To upgrade the operations center and controllers:

  1. Enable Quiet Restart to ensure that builds do not start automatically when the instance first comes online after the upgrade. Navigate to Manage Jenkins  Quiet Restart and select Stay in “quieting down” state when restarted.

  2. Verify all running builds are complete. Navigate to Manage Jenkins  Prepare for Shutdown and wait for any running builds to complete before you upgrade a controller.

  3. Take another backup of your instance.

  4. Upgrade the plugins and restart your controller immediately before you start the upgrade process.

  5. Visit the downloads page, select your desired distribution type, and complete the instructions:

    Java WAR file

    If your CloudBees CI instance is run with the command java -jar cloudbees-core-cm.war or java -jar cloudbees-core-oc.war, you can simply replace the jenkins.war file with the latest version.

    Ubuntu/Debian

    • Operations center: apt-get update && apt-get install cloudbees-core-oc

    • Controller: apt-get update && apt-get install cloudbees-core-cm

    Red Hat/CentOS

    The CloudBees CI on traditional platforms RPM installation package requires systemd. The majority of Linux distributions have adopted systemd, but if you attempt to use the RPM package to install CloudBees CI on traditional platforms on a Linux distribution that does not support systemd, the installation will appear to be successful, however the service will not be able to start.

    • Operations center: yum upgrade cloudbees-core-oc

    • Controller: yum upgrade cloudbees-core-cm

    RPM/YUM with High Availability (HA): The RPM package contains a post-installation script to ensure ownership on several files including $JENKINS_HOME. If HA is configured, I/O errors can occur. For example, when upgrading one node while another node is still running. For more information, refer to JENKINS-23273.

    To address this issue, you can skip this script by adding the property JENKINS_INSTALL_SKIP_CHOWN="true" under /etc/sysconfig/jenkins.
    OpenSUSE/SUSE Linux

    The CloudBees CI on traditional platforms RPM installation package requires systemd. The majority of Linux distributions have adopted systemd, but if you attempt to use the RPM package to install CloudBees CI on traditional platforms on a Linux distribution that does not support systemd, the installation will appear to be successful, however, the service will not be able to start.

    • Operations center: zypper install cloudbees-core-oc

    • Controller: zypper install cloudbees-core-cm

    Microsoft Windows

    To upgrade on Microsoft Windows servers, a new .zip file must be downloaded from the downloads page and installed.

    Servlet container

    Once a .war archive has been downloaded, follow the servlet container’s existing application deployment process.

    When using servlet containers, CloudBees CI sets the $JENKINS_HOME to the $APP_SERVER_USER/.jenkins/ folder. If the servlet container installation does not include write permissions to this folder for this user, which is sometimes done for security purposes, you must either grant appropriate permissions or override this setting by adding the -DJENKINS_HOME=$MY_JENKINSPATH argument in your servlet container startup. Refer to the servlet container’s documentation for how to add startup arguments.

    Tomcat

    Use environment variable CATALINA_OPTS to add:

    • -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true.

    • -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true which is needed for the Blue Ocean plugin.

    CloudBees recommends that you configure it in the script $CATALINA_BASE/bin/setenv.sh (Linux) or %CATALINA_BASE%\bin\setenv.bat (Microsoft Windows) that you will create to customize your application server.

    Custom container installations

    If you use a custom container, you can locate the cloudbees-core-oc.war or cloudbees-core-cm.war file in the deploy directory of your container. For example, /usr/local/jboss/server/default/deploy/cloudbees-core-cm.war is the location for a default JBoss installation.

  6. After your controllers are upgraded, you must manually upgrade the agent.jar file for any inbound agents that are not configured to upgrade the agent.jar file automatically. For more information, refer to Windows agent offline or unable to connect.

  7. Upgrade the plugins in your production environment again.

  8. Now that you are running the latest version of CloudBees CI and your plugins have been upgraded, CloudBees recommends you enable actionable build notifications in GitHub, Bitbucket, and Slack. This generally takes a few minutes and provides enhanced notifications for your developers with no Pipeline changes required. For more information, refer to Enable actionable build notifications in GitHub and Bitbucket and Set up actionable build notifications in Slack.

  9. Review articles in our Best Practices: CloudBees CI and apply changes accordingly.

Migrate to Java 11