Splitting a controller in CloudBees CI on traditional platforms

7 minute read

Splitting a controller involves the following tasks:

Be sure to read the entire document before starting this process and perform all the tasks to ensure a successful migration.

Initial steps

Before splitting a controller, do the following to reduce the quantity of data you need to move, ensure that your instance is in a healthy status, and have an environment that is as close as possible to the original one.

Review the Discard Build Policy

First, you should review the current strategy used to delete old builds. This will help to reduce the data involved in the splitting process. Please see the Deleting Old Builds - Best Strategy for Cleanup and disk space management article for more details. CloudBees recommends using the Global Build Discard Strategy (Manage Jenkin > Configure System > Global Build Discarders) as a policy to reduce the data Jenkins stores. This strategy is applied at the global level, so it is the quickest and easiest option to apply the same policy to all the jobs in a controller.

Review plugins

Next, CloudBees suggests that you review the plugin usage and remove any plugins you are not using. See How to determine if a plugin is in use.

Review the Jenkins controller health

CloudBees suggests that you install the Jenkins Health Advisor by CloudBees (formerly known as "CloudBees Jenkins Advisor"), which proactively notifies you of problems with your Jenkins-based environment. Jenkins Health Advisor by CloudBees identifies numerous issues before they affect your users, including security vulnerabilities, performance problems, and plugin version issues.

You should use this plugin to ensure the controller is healthy before splitting it. Otherwise, you might duplicate issues between multiple environments.

If you can’t use the plugin, CloudBees recommends opening a support case including the latest support bundle that you can find under $JENKINS_HOME/support, so that the CloudBees Support team can internally run the CloudBees Jenkins Advisor against your support bundle to identify potential issues with your Jenkins controller. However, using the Jenkins Health Advisor Plugin in your instance will help you to identify issues before they happen.

ONLY Linux - Prepare the new environment’s OS to work with CloudBees CI on traditional platforms

  1. Install the same Java version you are using in the original environment. Ideally, you will be running the latest JDK8 release that is available.

  2. Review the open files and max user processes in the OS by using the steps in this KB article.

  3. Disable Transparent Huge Pages by following the Huge pages section in the KB article CloudBees CI Performance Best Practices for Linux

  4. Ensure that the new filesystem is, at least, as fast as the one used by the source controller. If you use NFS, you should review the NFS Guide and ensure the NFS client’s recommended mount arguments are used. A minimum performance check can be run using the script from the KB article Required Data: IO issues on Linux and comparing the values in this table with the original environment where the source controller is located.

Install CloudBees CI on traditional platforms}

Download and install the binary that matches the version used by the source controller. Ensure the binary is the same one used by the source environment. Once the installation is successful, ensure that the Jenkins configuration is similar to the source environment. To do this, go to /etc/sysconfig(cloudbees-core-cm or /etc/default(cloudbees-core-cm, depending on the distribution, and ensure that the configuration is closed in both environments. Pay special attention if you see Java Arguments using DNS or IP addresses configured. For example, for -Dhudson.TcpSlaveAgentListener.hostName and -Djgroups.bind_addr you will need to reconfigure them with the new corresponding addresses.

Migrating data

Follow the steps below to migrate the data.

Start the source Jenkins controller in Quiet Mode

Before performing a backup of $JENKINS_HOME, CloudBees recommends starting your instance in Quiet Mode so there are no executors in use, and ideally, no jobs in the build queue at the time the instance backup is performed.

You can toggle Quiet Mode via the URLs $JENKINS_URL/quietDown and $JENKINS_URL/cancelQuietDown.

Export credentials

In Jenkins, $JENKINS_HOME/secrets/master.key is the secret used to encrypt all the credentials stored on disk. The newly created instances will have their own $JENKINS_HOME/secrets/master.key, so you must export and update the current credentials from the source to the destination controller.

The following scripts will help you to export your credentials at the System and Folder levels. However, credentials at the user level will not be exported. This is an infrequent use case, but you should consider that if there are any credentials created at the user level, you will need to manually recreate them.

Export system credentials

To migrate the system credentials:

  1. Download the export-credentials-system-level.groov script from GitHub.

  2. Run export-credentials-system-level.groovy in the Script Console on the source instance. It will put out an encoded message containing a flattened list of all system and folder credentials. Copy that encoded message.

You will use the encoded message later to update the System credentials in the new Jenkins controller.

Export folder credentials

To migrate the folder credentials:

  1. Download the export-credentials-folder-level.groovy script from GitHub.

  2. Run export-credentials-folder-level.groovy in the Script Console on the source instance. It will put out an encoded message containing a flattened list of all folder-level credentials. Copy that encoded message.

You will use the encoded message later to update the folder credentials in the new Jenkins controller.

Backup operation

You can perform the backup operation for $JENKINS_HOME either manually or using the CloudBees Backup plugin. This section explains how to perform a backup using the CloudBees Backup plugin.

  1. On the source controller, select New Item  Backup and Restore (project)

  2. In the Build section, select Take backup.

  3. When you configure the Take Backup section of this project, select What to back up? > System configuration and specify the following items in the Excludes list separated by commas.

    In the backup, regular expression for the System Configuration, nodes/ is excluded to avoid exporting any agent defined in the source client controller. If you want to keep all your dedicated agents defined in the source controller, remove nodes/ from the pattern list.

    • secret.key

    • license.xml

    • identity.key.enc

    • jgroups/

    • operations-center-cloud*

    • operations-center-client*

    • com.cloudbees.opscenter.client.plugin.OperationsCenterRootAction.xml

    • nodes/

    • Enable Omit controller key checkbox for not backing up $JENKINS_HOME/secrets/master.key

secrets/master.key,secret.key,license.xml,identity.key.enc,jgroups/,operations-center-cloud*,operations-center-client*,com.cloudbees.opscenter.client.plugin.OperationsCenterRootAction.xml,nodes/
▼

  1. Enable What to back up? > Build records if you would like to keep the build record in the migration.

  2. Enable What to back up? > Job configurations and specify the patterns to exclude from the backup separated by commas.

For example, if there is a structure like the one below, where you want to exclude folder-b, then the pattern to exclude would be jobs/folder-b.

Jenkins Root --> folder-a
             --> folder-b
▼

If the What to back up? > Job configurations is very complex for your use case, given the complexity of the anti-patterns for any jobs to be excluded, you can back up all the jobs, and later, once the restore process is completed, manually remove the jobs you would like from $JENKINS_HOME/jobs.

Restore operation

To restore the backup:

  1. Start the Jenkins process in the new environment.

  2. Enter the license for your instance ID. You must submit a support ticket and provide your instance ID to obtain a new license for the new client controller.

  3. Follow the instructions to create a first account.

  4. Install the suggested plugins.

  5. On the new controller, select the New Item  Backup and Restore (project).

  6. In the Build section, select Restore from backup.

  7. In Where to restore from ? select the backup created in Backup operation.

If you backed up all the jobs, you can remove any jobs you don’t want to keep from $JENKINS_HOME/jobs. When the restore is completed, restart the instance, and then use the filesystem to remove the unwanted jobs from $JENKINS_HOME/jobs.

Update system credentials

Download the update-credentials-system-level.groovy from GitHub.

Paste the encoded message output from the export-credentials-system-level.groovy script as the encoded variable value in the update-credentials-system-level.groovy script and execute it in the Script Console on the destination Jenkins. All the system-level credentials and domains from the source controller will now be updated to the system store of the destination controller.

The domains in the destination are not created automatically, and should be manually created before performing the update operation.

Update folder credentials

Download the update-credentials-folder-level.groovy from GitHub.

Paste the encoded message output from the export-credentials-folder-level.groovy script as the encoded variable value in the update-credentials-folder-level.groovy script and execute it in the Script Console on the destination controller. All the folder-level credentials and domains from the source controller will now be imported to each folder store on the destination controller.

Considerations

Agents

If you added /nodes to the backup regular expression for the System Configuration, you will have to recreate the nodes. Note that the reason why the nodes are excluded from the splitting process is because a node should not be connected to more than one controller, and they are already connected to the source controller.

Authentication

Depending on your environment and the authentication method used, your authentication might not work in the new client controller. For LDAP and Active Directory configurations, as long as there is connectivity between the controller and the authentication server, there should not be any problem.

For SAML configurations, however, you might need to reconfigure the client configuration in the client controller. If you can’t log in to the instance after the restart, you will need to disable security and reconfigure your Security Realm again.

WebHooks

WebHooks must be reconfigured in the new client controller.

If you are configuring Jenkins to automatically manage Bitbucket’s WebHooks, then these WebHooks are recreated when the corresponding job in Jenkins is saved. You can execute the Groovy script below under Manage Jenkins  Script Console to automatically recreate these WebHooks.

Jenkins.getInstanceOrNull().getItems(com.cloudbees.hudson.plugins.folder.AbstractFolder.class).each {
  println "Saving for WebHook recreation " + it.getFullDisplayName()
  it.save()
}
▼

If you use GitHub, you can automatically recreate all Webhooks for your Jenkins instance. Go to Manage Jenkins  Configure System  GitHub plugin  Advance  Re-registers hooks for all jobs.

Reconfigure the Jenkins URL

Navigate to Manage Jenkins  Configure System  Configure Security and update any settings as needed. Some settings (such as the Jenkins URL) are specific to each instance, and may need to be reconfigured for your destination instance.

One good example is cloud services such as Kubernetes cloud, EC2 cloud, or Docker cloud, in which we usually define the Jenkins URL. If the Jenkins URL field is populated, you will need to update this field to make the cloud work again.

Third-party integration

Third-party integrations might not work correctly in the new Jenkins controller, given that the Jenkins URL has changed. You should review all the URLs under Manage Jenkins  Configure System and make any necessary changes.

Third-party integrations might also fail for networking reasons. Make sure that your third-party tools can communicate with the new controller.

On modern cloud platforms