Updating a CasC bundle for the operations center

9 minute readScalabilityAutomation

This guide describes the bundle update process. When a bundle update is available, an alert is displayed in the Manage Jenkins screen.

Bundle update
Figure 1. Bundle update alert

CloudBees recommends that you complete the following steps to prepare for bundle update.

Check for bundle updates

You can also check for updates on the CloudBees Configuration as Code export and update screen on the Bundle update tab.

To check for bundle updates:

  1. Ensure you are signed in as a user with the Overall/Manage permission.

    This option is available to users with the Overall/Manage permission and it lets users with the Administrator role delegate this task. For more information, refer to Delegating Administration.
  2. Select Manage Jenkins in the left pane.

  3. Select CloudBees Configuration as Code export and update.

  4. Select the Bundle update tab. A blue bell appears next to the tab if an update is detected.

  5. Select Check for Updates.

    A bundle update is detected only when the version attribute in the bundle.yaml file has been incremented. If the version attribute is not incremented, the bundle update is not detected and cannot be applied.
    Check for updates
    Figure 2. Check for updates
  6. If bundle updates are available, you can select See the differences with the current version to view the differences between the current bundle and the new bundle.

    Bundle update file differences
    Figure 3. Bundle update file differences
  7. Select Back to Bundle update to return to the Bundle update tab.

  8. Select one of the following options to apply the new bundle:

    • Safe Restart: Performs a restart and applies the new bundle. CloudBees recommends this option because you can determine an appropriate time to restart and apply the updated bundle.

    • Reload Configuration: Applies the new bundle without a restart. This only reloads the bundle sections with changes (with new, updated, or deleted files). If there are changes to the variables.yaml file, a full reload is performed.

      The Reload Configuration: option is called a "hot reload". For more information on when a hot reload is appropriate, refer to Using hot reload on a configuration bundle.

      New bundle version
      Figure 4. New bundle version
      You can optionally select Force Reload to reapply a bundle that is already installed. This option can be used to reset changes in the configurations managed by CasC. The new configuration is not going to be removed.

If you want to view the update log, refer to Viewing the operations center update log.

Updating bundles with branch and tag support

CloudBees CasC Server plugin 2.0 introduces bundle branch and tag support. The plugin update changes the folder structure within JENKINS_HOME which stores CasC bundles. In older versions of the CloudBees CasC Server plugin, the folder structure within JENKINS_HOME was organized by bundle name with one folder per bundle. Now the folder structure within JENKINS_HOME can store multiple versions of each bundle. This makes it easy to maintain bundles for different environments or releases.

Migrating bundles

Due to changes to the JENKINS_HOME folder structure with the CloudBees CasC Server plugin 2.0 update, your existing bundles must be migrated. Bundle migration allows you to select bundles ready for update and assign them to the approprite branch within a controller. During the migration process, the original bundle configuration remains until the migration is complete. This prevents the loss of data and misconfiguration of the controller. Bundle migration is handled manually or automatically.

Before running the migrating process, do the following:

  • Generate a list of all repositories without the 'Default version' configuration.

  • Generate a list of all controllers using a bundle that is not available in the operations center

  • Apply the`Default version` configuration to bundles from a source control management repository (SCM).

  • Update the items.yaml in your operations center bundle.

To generate a list of all repositories without the Default version configuration and controllers using a bundle that is not available in the operations center:

  • Run the following Groovy script in the Script Console.

import com.cloudbees.opscenter.server.casc.config.remotes.RemoteBundles;
import com.cloudbees.opscenter.server.casc.config.remotes.SCMBundleRetriever;
import com.cloudbees.opscenter.server.model.ConnectedMaster;
import com.cloudbees.opscenter.server.casc.config.ConnectedMasterCascProperty;

def remotes = ExtensionList.lookupSingleton(RemoteBundles.class).getBundles();
def remotesWithoutDefVersion = [] as LinkedList
remotes.each{
    if (it.getRetriever() instanceof SCMBundleRetriever) {
        def defaultVersion = it.getRetriever().getDefaultVersion()
        if (!(defaultVersion?.trim())) {
            remotesWithoutDefVersion.add(it.getName());
        }
    }
}
println "CasC Controller Bundle location check"
println "====================================="
if (remotesWithoutDefVersion) {
    println "The following remote configurations  do not have a default version:"
    remotesWithoutDefVersion.each{
        println "- " + it
    }
    println "You can fix this in Manage Jenkins > Configure System > Load CasC Bundles."
} else {
  println "All your CasC Controller Bundle locations have configured the Default Version or are Local Folders"
}
println ""
println "CasC Controller Bundle existence check"
println "======================================"
def bundlesInDisk = Jenkins.get().getRootDir().toPath().resolve("cb-casc-bundles-store").toFile().list()
def bundlesNotInDisk = [:]
Jenkins.get().getAllItems(ConnectedMaster.class).each{
    def property = it.getProperties().get(ConnectedMasterCascProperty.class);
    if (property != null) {
        if(!bundlesInDisk.contains(property.getBundle())) {
            bundlesNotInDisk[it.getFullName()] = property.getBundle()
        }
    }
}
if (bundlesNotInDisk.size() > 0) {
    bundlesNotInDisk.each{
        println "Controller $it.key has assigned the bundle $it.value but it cannot be found. Please assign a valid bundle"
    }
    println "Remember all your controllers need a valid bundle for Branch/Tag support"
} else {
    println "All your controllers are using existent bundles"
}

To apply the Default version configuration:

  1. Go to the operations center dashboard and select Manage Jenkins  Manage Jenkins  Configure System.

  2. Navigate to Configuration as Code bundle location.

  3. Select Configure.

  4. Set the value for Default version for each configured repository. This new value should be the branch your instance is currently using to checkout the CasC bundles.

  5. Select Save.

Manual bundle migration

When there are multiple versions of the bundle from different branches, an alert is displayed in the Manage Jenkins screen.

bundle migration alert
Figure 5. Bundle migration alert
You can proceed with the manual migration by selecting the link to the Manual branch migration screen or selecting Manual Branch Migration on the left side of the operations center dashbaord.

To migrate a bundle:

  1. On the left side of the Dashboard, select Manual Branch Migration. The Manual Branch Migration screen displays.

  2. Migrate bundles by doing one of the following:

    • Select Action needed. For each bundle listed, select the appropriate branch or tag to assign it to that bundle. Selecting the branch or tag updates all the controllers using the older version of the bundle.

      bundles needing manual migration
      Figure 6. Bundles needing manual migration
    • Select Advanced. Create and upload a yaml file to assign different branches or tags to a controller using the same bundle.

      bundle migration advanced tab
      Figure 7. Bundle migration - Advanced tab

      Use the following format for the yaml input:

      markAsComplete: true
      controllers:
          full-path-controller-1: branch1/bundle1
          full-path-controller-2: branch2/bundle1
          full-path-controller-3: another-branch/another-bundle
  3. Select Save.

Selecting Discard and complete will mark the migration process as complete and the discarded bundles will retain their legacy configuration. However, you can still migrate the discarded bundles.

Automatic bundle migration

Automatic bundle migration occurs when the bundle contains only one branch or the Default version has been configured. Select Migrated on the Manual Branch Migration screen to view all bundles automatically migrated.

Bundles migrated tab
Figure 8. Bundles automatically migrated

Migrating discarded bundles

Bundles discarded during the initial migration process can still be migrated.

To migrate a previously discarded bundle:

  1. For each controller do the following:

    1. Go to JENKINS_URL/job/CONTROLLER/configure and select a bundle.

    2. Select Save.

  2. Remove the temporary bundle source.

    1. Go to Manage Jenkins  Configure System  Configuration as Code bundle location

    2. Select Configure.

    3. Remove the remote named as legacyBundleConfiguration.

    4. Select Save.

  3. Go to JENKINS_HOME/legacyBundleConfiguration to remove the physical folder with the legacy configurations.

  4. (Optional) If the user has configured the default bundle:

    1. Go to Manage Jenkins  CloudBees Configuration as Code bundles  General settings tab

    2. Select the new default bundle.

    3. Select Save.

  5. From the JENKINS_HOME folder, remove any existing files called retriever-branch.*, except on .retriever-branch.migrated.

    • JENKINS_HOME/retriever-branch.pendingRetriever

    • JENKINS_HOME/retriever-branch.pendingMatch

    • JENKINS_HOME/retriever-branch.pendingDefaultBundle

Revert bundle migration

Revert a bundle migration if the following occurs:

  • An error is detected during the migration process that prevents the migration from restarting.

  • Jenkins crashes and causes instability with JENKINS_HOME and no backup was performed.

To revert a bundle migration:

  1. Run one of the following commands:

    • Delete JENKINS_HOME/.retriever-branch.migrated

    • Delete JENKINS_HOME/.retriever-branch.pendingRetriever

    • Delete JENKINS_HOME/.retriever-branch.pendingDefaultBundle

    • Delete JENKINS_HOME/.retriever-branch.pendingMatch

  2. Remove the physical folder with the legacy configurations from JENKINS_HOME/legacyBundleConfiguration.

  3. Locate an entry for legacyBundleConfiguration on JENKINS_HOME/com.cloudbees.opscenter.server.casc.config.remotes.RemoteBundles.xml and remove it from the file.

Validating bundles prior to update

Before updating bundles and adding them to the operations center, you can validate any changes made to them. Bundle prevalidation ensures that changes can be successfully applied to prevent controller outages. Some of the validations include:

  • There are no syntax errors.

  • Effective bundles can be calculated correctly.

  • Effective bundles are valid against available controllers.

For more information on validating bundles refer to Troubleshooting CasC for the operations center or Troubleshooting CasC for controllers.

Validating bundles in your CLI

Run the following command to validate your bundle(s):

java -jar jenkins-cli.jar casc-pre-validate-bundle < bundle.zip (1)
1 The bundle.zip contains the bundles you want to validate.

Validating bundles using HTTP endpoint

Make a POST request using JENKINS_URL/casc-bundle/pre-validate-bundle uploads the provided bundles to the operations center without making them available, then applies the CasC bundle inheritance, and then validates the provided bundles. The output is the validation report without having to invoke any other endpoint.

The expected input is a .zip file that contains the bundles to validate (there is no need to include all bundles in the inhertance chain if they are available in the operations center). The output is not the effective bundle in .zip format, but the validation report.

An example of the Post request:

curl -s -H 'Accept: application/json' -H 'Content-Type: application/zip;charset=utf-8' --user [USER] --data-binary @[PATH_TO_ZIP] -XPOST $JENKINS_URL/casc-bundle/pre-validate-bundle

A report is generated once the bundles are validated. The following is an example of validated bundle report.

{"validations": [
        {
        "bundle": "bundle-2", (1)
        "result":         {
            "valid": true, (2)
            "structureValidations": [],
            "controllerValidations":             [
                                {
                    "controller": "landon-controller",
                    "controllerStatus": "ONLINE",
                    "validations": []
                },
                                {
                    "controller": "ccontroller-2",
                    "controllerStatus": "ONLINE",
                    "validations": []
                }
            ]
        }
    },
        {
        "bundle": "bundle-1",
        "result":         {
            "valid": false,
            "structureValidations": [            {
                "level": "ERROR", (3)
                "validationCode": "BUNDLE_CONTENT",
                "message": "[CONTVAL] - Missing Jenkins Configuration as Code file specified in the bundle.yaml file." (4)
            }],
            "controllerValidations": []
        }
    }
]}
1 "bundle": Bundle name
2 "valid": 'true' indicates the bundle validation was a success. 'false' indicates the validation failed.
3 "level": If the bundle validation fails, this field displays with an 'ERROR'.
4 "message": Failed bundle validation error message.

Validating bundles with GitHub checks

Instead of using CLI or HTTP endpoints to validate bundles, you can also use GitHub Checks. When a pull request (PR) is generated against a specific bundle, prevalidation checks are run, and the results are displayed on the Checks tab of the PR.

The GitHub App authentication for your GitHub Organization must be enabled to use the Checks feature.
Bundle validation in the GitHub checks tab
Figure 9. Bundle validation in the GitHub checks tab

The Checks tab displays prevalidation details for each branch, including error messages.

Configuring the bundle update recurrence period

After the initial configuration on startup, a background process automatically checks for bundle updates every twenty minutes.

Optionally, you can customize this by setting the Java system property com.cloudbees.opscenter.client.casc.ConfigurationUpdaterTask.recurrencePeriod to the desired interval, in minutes.

For example, to check for updates every five minutes, use the following startup command:

java -Dcore.casc.config.bundle=$JENKINS_HOME/casc-bundle-link.yml \
 -Dcom.cloudbees.opscenter.client.casc.ConfigurationUpdaterTask.recurrencePeriod=5 \
 -jar jenkins.war

Using CasC gives you great power and flexibility, which introduces the possibility for errors that could impact your installation. Please review the information below to help guide you as you evolve your CasC bundle.

Risks when updating the bundle.yaml file

A bundle update is detected only when the version attribute in the bundle.yaml file has been incremented. If the version attribute is not incremented, the bundle update is not detected, and cannot be applied.

Risks when updating the plugins.yaml file

You can install CloudBees Assurance Program (CAP) Tier 1 and Tier 2 plugins by adding the plugins to the plugins.yaml file.

CAP Tier 1 and Tier 2 plugins

To get the list of CloudBees Tier 1 and Tier 2 plugins:

  1. Navigate to CloudBees CI Plugins.

  2. To verify that your plugin is a CAP Tier 1 or Tier 2 plugin, you can either:

    • Search for a specific plugin and filter based on Verified or Proprietary (Tier 1) or Compatible (Tier 2).

    • Scroll to the bottom of the page and select All Tier 1 plugins to browse all Tier 1 plugins and then filter by Tier.

For more information on the CAP and how CloudBees classifies plugins, refer to CloudBees plugin support policies.

Uninstalling a plugin

There are no "delete/uninstall” operations when using CasC, and you cannot uninstall plugins by removing the plugins from the plugins.yaml file.

To uninstall a plugin:

  1. Uninstall the plugin in the UI.

  2. Before restarting, remove the plugin IDs from the plugins.yaml file. Otherwise, the plugins will be reinstalled.

Risks when updating the jenkins.yaml file

This file can be edited to change or add Jenkins configurations. There are a wide variety of possible combinations, depending on the plugins that are configured as code. This file is managed by the Jenkins LTS - Configuration as Code (JCasC) plugin. Therefore, all Jenkins LTS documentation, reported bugs, community blogs, Gitter chats, etc., apply to this file.

The bulk of the content in this file is defined by the various plugins you have installed with only a small portion of it defined by the Jenkins core itself. However, there are a few general items to consider when editing the file:

  • Removing a configuration from this file does not remove the configuration from an existing controller or operations center instance.

  • Adding configurations requires the underlying plugin that the configuration references to be installed. Jenkins startup will fail otherwise.

  • Backward compatibility depends on the specific plugins that are being configured. If a plugin changes its configuration in an incompatible way, then the controller or the operations center will fail to start until the jenkins.yaml file is adapted, or the section in the file that is related to the plugin is removed.

  • This file may contain data which is tied to a specific CloudBees instance, such as encoded secrets, the operations center connection details, and others, so a jenkins.yaml is, in general, suitable for a single CloudBees controller instance. This might lead to indecipherable secrets, controller to operations center connection failing, and many more things depending on the specific configuration.

Using hot reload on a configuration bundle

CasC allows you to reload a configuration file without restarting the instance. Since the CasC API plugin manages plugin updates, validating a hot reload is a more complex process. A bundle reload is not always possible. In the UI, if the bundle hot reload is not possible, the Reload Configuration button does not appear on the Bundle update tab of the CloudBees Configuration as Code export and update screen.

If plugin upgrades are in the updated bundle, Jenkins must be restarted.

If a hot reload is possible, the following operations are automatically performed:

  • Install any new plugins.

  • Call the API on configuration as code to reload the Jenkins configuration.

Configuration bundles that do not include plugin updates can be reloaded without an instance restart. This feature allows you to update Jenkins configurations or install new plugins dynamically.