Splitting a controller in CloudBees CI on modern cloud platforms

7 minute read

Splitting a controller involves the following tasks:

Be sure to read the entire document before starting this process and perform all the tasks to ensure a successful migration.

Initial steps

Before splitting a controller, do the following to reduce the quantity of data you need to move, ensure that your instance is in a healthy status, and have an environment that is as close as possible to the original one.

Review the Discard Build Policy

First, you should review the current strategy used to delete old builds. This will help to reduce the data involved in the splitting process. Please see the Deleting Old Builds - Best Strategy for Cleanup and disk space management article for more details. CloudBees recommends using the Global Build Discard Strategy (Manage Jenkins  Configure System  Global Build Discarders) as a policy to reduce the data Jenkins stores. This strategy is applied at a global level, so it is the quickest and easiest option to apply the same policy to all the jobs in a controller.

Review plugins

Next, CloudBees suggests that you review the plugin usage and remove any plugins you are not using. See How to determine if a plugin is in use.

Review the Jenkins Controller health

CloudBees suggests that you install the Jenkins Health Advisor by CloudBees (formerly known as "CloudBees Jenkins Advisor"), which proactively notifies you of problems with your Jenkins-based environment. Jenkins Health Advisor by CloudBees identifies numerous issues before they affect your users, including security vulnerabilities, performance problems, and plugin version issues.

You should use this plugin to ensure the controller is healthy before splitting it. Otherwise, you might duplicate issues between multiple environments.

If you can’t use the plugin, CloudBees recommends opening a support case including the latest support bundle that you can find under $JENKINS_HOME/support, so that the CloudBees Support team can internally run the CloudBees Jenkins Advisor against your support bundle to identify potential issues in your Jenkins controller. However, using the Jenkins Health Advisor Plugin in your instance will help you identify issues before they happen.

Create the new managed controller

In the operations center, create a new managed controller (New Item > Managed Controller). In the managed controller configuration, ensure that the configuration is close to the source managed controller. You will need to review all the fields and ensure that both have a similar configuration. Pay special attention to the following fields, which usually should match with the source managed controller fields:

  • Docker image

  • Jenkins Controller Memory in MB

  • Jenkins Controller CPUs

  • Jenkins Command Line arguments

  • Java Options

  • Environment Variables

  • System Properties

  • Termination Grace Period

  • Liveness Period

  • Liveness Timeout

  • Readiness Initial Delay

  • Filesystem group

  • Image pull secrets

The Docker image used in the new managed controller must match the source managed controller, otherwise it will not be a migration, but a migration and an upgrade at the same time. It is important that you only perform the migration so that you do not have to deal with possible upgrade issues in addition to the migration.

The restore operation could take minutes or hours, depending on the quantity of data being moved and the speed of your storage system. For this reason, and only until the restore operation is completed, you should adjust the fields below to ensure that Kubernetes does not restart the controller in the middle of the backup for not running properly. The following values ensure your restore process will complete if it takes less than 5 hours, which should be enough time for most cases.

  • Liveness Initial Delay → 18000

  • Readiness Timeout → 30

  • Readiness Failure Threshold → 600

In the sections Java Options, Environment Variables, and System Properties, you might need to change specific values related to Kubernetes services or the Jenkins URL.

Until the restore process is performed, CloudBees suggests that you configure On-controller executors > Override # of executors with the number 1. This enables the controller executor instead of using a Kubernetes Pod template to make the restoration process easier, less error prone, and faster.

Remember to disable On-controller executors once the restoration process is performed.

Migrating data

Follow the steps below the migrate the data.

Start the source Jenkins controller in Quiet Mode

Before you back up $JENKINS_HOME, CloudBees recommends starting your instance in Quiet Mode so there are no executors in use, and ideally, no jobs in the build queue at the time the instance backup is performed.

You can toggle Quiet Mode via the URLs $JENKINS_URL/quietDown and $JENKINS_URL/cancelQuietDown.

Export credentials

In Jenkins, $JENKINS_HOME/secrets/master.key is the secret used to encrypt all the credentials stored on disk. The newly created instances have their own $JENKINS_HOME/secrets/master.key, so you must export and update the current credentials from the source to the destination controller.

The following scripts will help you to export your credentials at the System and Folder levels. However, user-level credentials are not exported. It is an infrequent use case, but you should consider if you have any user-level credentials, you will need to manually recreate them.

Export system credentials

To migrate the system credentials:

  1. Download the export-credentials-system-level.groov script on GitHub.

  2. Run export-credentials-system-level.groovy in the Script Console on the source instance. It will put out an encoded message containing a flattened list of all system and folder credentials. Copy that encoded message.

You will use the encoded message later to update the system credentials in the new Jenkins controller.

Export folder credentials

To migrate the folder credentials:

  1. Download the export-credentials-folder-level.groovy script on GitHub.

  2. Run export-credentials-folder-level.groovy in the Script Console on the source instance. It will put out an encoded message containing a flattened list of all credentials at the folder level. Copy that encoded message.

You will use the encoded message later on to update the folder credentials in the new Jenkins controller.

Backup operation

You can perform the backup operation for $JENKINS_HOME either manually or using the CloudBees Backup plugin. This section explains how to perform a backup using the CloudBees Backup plugin.

  1. On the source controller, select New Item  Backup and Restore (project)

  2. In the Build section, select Take backup.

  3. When you configure the Take Backup section of this project, select What to back up? > System configuration and specify the following items in the Excludes list separated by commas.

In the backup, regular expression for the System Configuration, nodes/ is excluded to avoid exporting any agent defined in the source managed controller. If you want to keep all your dedicated agents defined in the source controller, then remove nodes/ from the pattern list.

  • secret.key

  • license.xml

  • identity.key.enc

  • jgroups/

  • operations-center-cloud*

  • operations-center-client*

  • com.cloudbees.opscenter.client.plugin.OperationsCenterRootAction.xml

  • nodes/

  • Enable Omit controller key checkbox for not backing up $JENKINS_HOME/secrets/master.key

secrets/master.key,secret.key,license.xml,identity.key.enc,jgroups/,operations-center-cloud*,operations-center-client*,com.cloudbees.opscenter.client.plugin.OperationsCenterRootAction.xml,nodes/
▼

  1. Enable What to back up? > Build records if you want to keep the build record in the migration.

  2. Enable What to back up? > Job configurations and specify the patterns to exclude from the backup, separated by commas.

For example, if there is a structure like the one below, where you want to exclude folder-b, then the pattern to exclude would be jobs/folder-b.

Jenkins Root --> folder-a
             --> folder-b
▼

If the What to back up? > Job configurations is very complex for your use case, given the complexity of the anti-patterns for any jobs to be excluded, you can back up all the jobs, and later, once the restore process is completed, manually remove the jobs you would like from $JENKINS_HOME/jobs.

Restore operation

Before restoring the data, you must start the destination managed controller. The first time a managed controller is started, you are prompted to install suggested plugins or to select plugins to install. Select install suggested plugins, which allows you to use the restore operation without any issues.

Once the main dashboard of the destination managed controller is available, you can create a restore job.

To create a restore job:

  1. On the new controller, select the New Item  Backup and Restore (project).

  2. In the Build section, select Restore from backup.

  3. In Where to restore from ? select the backup created in Backup operation.

If you backed up all the jobs, you can remove any jobs you don’t want to keep from $JENKINS_HOME/jobs. When the restore is completed, restart the instance, and then use the filesystem to remove the unwanted jobs from $JENKINS_HOME/jobs.

Update system credentials

Download the update-credentials-system-level.groovy on GitHub.

Paste the encoded message output from the export-credentials-system-level.groovy script as the value in the encoded variable in the update-credentials-system-level.groovy script and execute it in the Script Console on the destination Jenkins. All the source controller’s system-level credentials and domains will now be updated to the system store of the destination controller.

Update folder credentials

Download the update-credentials-folder-level.groovy on GitHub.

Paste the encoded message output from the export-credentials-folder-level.groovy script as the value in the encoded variable in the update-credentials-folder-level.groovy script and execute it in the Script Console on the destination controller. All the source controller’s folder-level credentials and domains will now be imported to the destination controller’s folder store.

Considerations

Review the following important considerations.

Agents

If you added /nodes to the backup regular expression for the System Configuration, you will have to recreate the nodes. Note that the reason why the nodes are excluded from the splitting process is because a node should not be connected to more than one controller, and they are already connected to the source controller.

Authentication

Depending on your environment and the authentication method used, your authentication might not work in the new client controller. For LDAP and Active Directory configurations, as long as there is connectivity between the controller and the authentication server, there should not be any problem.

For SAML configurations, however, you might need to reconfigure the client configuration in the client controller. If you can’t log in to the instance after the restart, you will need to disable security and reconfigure your Security Realm again.

WebHooks

WebHooks must be reconfigured in the new managed controller.

If you are configuring Jenkins to automatically manage Bitbucket’s WebHooks, then these WebHooks are recreated when the corresponding job in Jenkins is saved. You can execute the Groovy script below under Manage Jenkins > Script Console to automatically recreate these WebHooks.

Jenkins.getInstanceOrNull().getItems(com.cloudbees.hudson.plugins.folder.AbstractFolder.class).each {
  println "Saving for WebHook recreation " + it.getFullDisplayName()
  it.save()
}
▼

If you use GitHub, you can automatically recreate all WebHooks for your Jenkins instance. Go to Manage Jenkins  Configure System  GitHub plugin  Advance  Re-register hooks for all jobs.

Reconfigure the Jenkins URL

Navigate to Manage Jenkins  Configure System  Configure Security and update any settings as needed. Some settings, such as the Jenkins URL, are specific to each instance, and may need to be reconfigured for your destination instance.

One good example is cloud services such as Kubernetes cloud, EC2 cloud, or Docker cloud, in which the Jenkins URL is usually defined. If the Jenkins URL field is populated, you will need to update this field to make the cloud work again.

Third-party integration

Third-party integrations might not work correctly in the new Jenkins controller given that the Jenkins URL has changed. You should review all the URLs under Manage Jenkins  Configure System and make any necessary updates.

Third-party integrations might also fail for networking reasons. Make sure that your third-party tools can communicate with the new controller.

Introduction
On traditional platforms