Updating a CasC bundle for the operations center

This guide describes the bundle update process. When a bundle update is available, an alert is displayed in the Manage Jenkins screen.

Figure 1. Bundle update alert

Check for bundle updates

You can also check for updates on the CloudBees Configuration as Code export and update screen on the Bundle update tab.

To check for bundle updates:

Ensure you are signed in as a user with the Overall/Manage permission.

This option is available to users with the Overall/Manage permission, and it lets users with the Administrator role delegate this task. For more information, refer to Delegating Administration.

Select Manage Jenkins in the left pane.
Select CloudBees Configuration as Code export and update.
Select the Bundle update tab. A blue bell appears next to the tab if an update is detected.

Select Check for Updates.

A bundle update is detected only when the version attribute in the bundle.yaml file has been incremented. If the version attribute is not incremented, the bundle update is not detected and cannot be applied. However, when using the CasC Bundle Retriever for the operations center, you can activate auto-versioning, which replaces the version attribute with the commit hash obtained from your source control management (SCM) system.

For modern platforms, after you issue the kubectl apply command and the updated CasC bundle is applied to the Kubernetes cluster, it may take approximately 60 seconds for the updated bundle to be detected by the operations center.

Figure 2. Check for updates

The Bundle update tab displays bundle validation messages. Each CasC bundle validation message includes the type of validation, the total number of validation messages, a general validation code, and a descriptive message.

View Bundle update tab with validation warning message

View Bundle update tab with no validation warning/error message

View Bundle update tab with validation messages in quiet mode
If bundle updates are available, you can select See the differences with the current version to view the differences between the current bundle and the new bundle.

Figure 3. Bundle update file differences
Select Back to Bundle update to return to the Bundle update tab.

Select one of the following options to apply the new bundle:

Safe Restart: Performs a restart and applies the new bundle. CloudBees recommends this option because you can determine an appropriate time to restart and apply the updated bundle.

Reload Configuration: Applies the new bundle without a restart. This only reloads the bundle sections with changes (with new, updated, or deleted files). If there are changes to the variables.yaml file, a full reload is performed.

The Reload Configuration: option is called a "hot reload". For more information on when a hot reload is appropriate, refer to Using hot reload on a configuration bundle.

Figure 4. New bundle version

You can optionally select Force Reload to reapply a bundle that is already installed. This option can be used to reset changes in the configurations managed by CasC. The new configuration is not going to be removed.

If you want to view the update log, refer to Viewing the operations center update log.

Updating bundles with branch and tag support

CloudBees CasC Server plugin 2.0 introduces bundle branch and tag support. The plugin update changes the folder structure within JENKINS_HOME that stores CasC bundles. In older versions of the CloudBees CasC Server plugin, the folder structure within JENKINS_HOME was organized by bundle name with one folder per bundle. Now, the folder structure within JENKINS_HOME can store multiple versions of each bundle. This makes it easy to maintain bundles for different environments or releases.

Migrating bundles

Due to folder structure changes within JENKINS_HOME your existing bundles must be migrated. Bundle migration allows you to select bundles ready for update and assign them to the appropriate branch within a controller. During the migration process, the original bundle configuration remains until the migration is complete. This prevents the loss of data and misconfiguration of the controller. You can run the bundle migration process within the operations center or outside of the operations center.

If describing controllers as items, also specify the branch in the items.yaml file post migration as shown in the example below.

removeStrategy:
  items: "none"
  rbac: "sync"
items:
  - kind: managedController
    name: payments-team
    configuration:
      kubernetes:
        memory: 4096
        cpus: 2.0
        disk: 2
    properties:
      - configurationAsCode:
          bundle: main/payments-team▼

Before running the migration process, apply the defaultVersion configuration to bundles in a source control management (SCM) repository.

To configure defaultVersion value in jenkins.yaml:

unclassified:
  bundleStorageService:
    activated: true
    purgeOnDeactivation: true
    pollingPeriod: 12
    activeBundle:
      name: "initial-remote"
      polling: false
      retriever:
        SCM:
          defaultVersion: "default-branch" (1)
          scmSource:
            git:
              id: "bf0d6af0-87a3-47bc-bd47-b3341d812cb9"
              remote: "https://github.com/cloudbees/pulsar-casc-bundles"
              traits:
                - "gitBranchDiscovery"▼

1	Set the `defaultVersion` value.

To configure defaultVersion in the operations center:

These steps can only be performed if you have not upgraded to CloudBees CasC Server plugin 2.0.

From the dashboard, select Manage Jenkins Manage Jenkins System.
Navigate to Configuration as Code bundle location.
Select Load CasC bundles, and then select Configure.
Set the value for defaultVersion for each configured repository. This new value should be the branch your instance is currently using to check out the CasC bundles.
Select Save.

For reporting purposes, generate a list of all repositories without the defaultVersion configuration:

Run the following Groovy script in the Script Console.

import com.cloudbees.opscenter.server.casc.config.remotes.RemoteBundles;
import com.cloudbees.opscenter.server.casc.config.remotes.SCMBundleRetriever;
def remotes = ExtensionList.lookupSingleton(RemoteBundles.class).getBundles();
def remotesWithoutDefVersion = [] as LinkedList
remotes.each{
  if (it.getRetriever() instanceof SCMBundleRetriever) {
    def defaultVersion = it.getRetriever().getDefaultVersion()
    if (!(defaultVersion?.trim())) {
      remotesWithoutDefVersion.add(it.getName());
    }
  }
}
if (remotesWithoutDefVersion) {
  println "The following remote configurations read the bundles from SCM and they do not have a default version:"
  remotesWithoutDefVersion.each{
    println "- " + it
  }
  println "You can fix this in Manage {JOSS} > Configure System > Load CasC Bundles."
} else {
  println "All your CasC Controller Bundle locations have configured the Default Version or are Local Folders"
}▼

Bundle migration within the operations center

In the operations center the bundle migration process can be handled manually or automatically.

Manual bundle migration

When there are multiple versions of the bundle from different branches, an alert is displayed on the Manage Jenkins screen.

Figure 5. Bundle migration alert

You can proceed with the manual migration by selecting the link to the Manual Branch Migration screen or selecting Manual Branch Migration on the left side of the operations center dashboard.

To migrate a bundle:

On the left side of the Dashboard, select Manual Branch Migration. The Manual Branch Migration screen displays.
Migrate bundles by doing one of the following:
- Select Action needed. For each bundle listed, select the appropriate branch or tag to assign it to that bundle. Selecting the branch or tag updates all the controllers using the older version of the bundle.
  
  Figure 6. Bundles needing manual migration
- Select Advanced. Create and upload a YAML file to assign different branches or tags to a controller using the same bundle.
  
  Figure 7. Bundle migration - Advanced tab
  
  Use the following format for the YAML input:
  markAsComplete: true controllers: full-path-controller-1: branch1/bundle1 full-path-controller-2: branch2/bundle1 full-path-controller-3: another-branch/another-bundle
  ▼
Select Save.

Selecting Discard and complete will mark the migration process as complete and the discarded bundles will retain their legacy configuration. However, you can still migrate the discarded bundles.

Automatic bundle migration

Automatic bundle migration occurs when the bundle contains only one branch or the Default version has been configured. Select Migrated on the Manual Branch Migration screen to view all bundles automatically migrated.

Figure 8. Bundles automatically migrated

Bundle migration outside of the operations center

You can migrate bundles without having to access the Manual branch migration page in the operations center.

Make sure all bundle locations have a defaultVersion branch set in the operations center.
Upgrade to a product version that includes CloudBees CasC Server plugin 2.0 for operations center or newer (first released in 2.401.1.3).
Restart your operations center instance to enable branch and tag support.

Go to your items.yaml export file. It should display as follows:

items:
- kind: managedController
  name: basic-controller
  properties:
  - configurationAsCode:
      bundle: main/basic-controller▼

Update the items.yaml file in the operations center with branch names.

If you are using an SCM, use the Name in the items.yaml export file.

If you are using a local folder, use the Name from either the items.yaml export file or the bundleStorageService.

The name of the bundleStorageService can be found in two places:

The Name displayed in the Manage Jenkins Configure System Configuration as Code bundle location Load CasC Bundles setting.
The Name displayed in the bundleStorageService configuration of the jenkins.yaml file.

The following example shows the Name value in the bundleStorageService configuration within a jenkins.yaml file.

unclassified:
  bundleStorageService:
    activated: true
    bundles:
      - name: "test" (1)
        polling: false
        retriever:
          localFolderRetriever:
            localFolder: /path/to/bundles▼

1	The Name value in the `bundleStorageService` configuration

Upgrade controllers to a product version that includes CloudBees CasC Client plugin 2.0 for controllers or newer (first released in 2.401.1.3).
Restart your instance to enable branch and tag support.

Migrating discarded bundles

Bundles discarded during the initial migration process can still be migrated.

To migrate a previously discarded bundle:

For each controller do the following:
1. Go to JENKINS_URL/job/CONTROLLER/configure and select a bundle.
2. Select Save.
Remove the temporary bundle source.
1. Go to Manage Jenkins Configure System Configuration as Code bundle location.
2. Select Configure.
3. Remove the remote named as legacyBundleConfiguration.
4. Select Save.
Go to JENKINS_HOME/legacyBundleConfiguration to remove the physical folder with the legacy configurations.
(Optional) If the user has configured the default bundle:
1. Go to Manage Jenkins CloudBees Configuration as Code bundles General settings tab.
2. Select the new default bundle.
3. Select Save.
From the JENKINS_HOME folder, remove any existing files called retriever-branch.*, except on .retriever-branch.migrated.
- JENKINS_HOME/retriever-branch.pendingRetriever
- JENKINS_HOME/retriever-branch.pendingMatch
- JENKINS_HOME/retriever-branch.pendingDefaultBundle

Revert bundle migration

Revert a bundle migration if the following occurs:

An error is detected during the migration process that prevents the migration from restarting.
Jenkins crashes and causes instability with JENKINS_HOME and no backup was performed.

To revert a bundle migration:

Run one of the following commands:
- Delete JENKINS_HOME/.retriever-branch.migrated
- Delete JENKINS_HOME/.retriever-branch.pendingRetriever
- Delete JENKINS_HOME/.retriever-branch.pendingDefaultBundle
- Delete JENKINS_HOME/.retriever-branch.pendingMatch
Remove the physical folder with the legacy configurations from JENKINS_HOME/legacyBundleConfiguration.
Locate an entry for legacyBundleConfiguration on JENKINS_HOME/com.cloudbees.opscenter.server.casc.config.remotes.RemoteBundles.xml and remove it from the file.

Validating bundles prior to update

Before updating bundles and adding them to the operations center, you can validate any changes made to them. Bundle prevalidation ensures that changes can be successfully applied to prevent controller outages. Some of the validations include:

There are no syntax errors.
Effective bundles can be calculated correctly.
Effective bundles are valid against available controllers.

For more information on validating bundles refer to Troubleshooting CasC for the operations center or Troubleshooting CasC for controllers.

When a bundle update adds a property to a YAML file, and a second bundle update removes the property from the YAML file, the property remains in the initial version of the bundle. To remove the property value from the instance, set the property value to its default value. For example:

Bundle version 1 has a systemProperties: example.
Bundle version 2 removes the property from the YAML file.

The systemProperties: example remains in Bundle version 1. To remove that property from Bundle version 1, set systemProperties: “”.

Validating bundles in your CLI

Run the following command to validate your bundle(s):

java -jar jenkins-cli.jar casc-pre-validate-bundle < bundle.zip (1)▼

1	The bundle.zip contains the bundles you want to validate.

Validating bundles using HTTP endpoint

Make a POST request using JENKINS_URL/casc-bundle/pre-validate-bundle uploads the provided bundles to the operations center without making them available, then applies the CasC bundle inheritance, and then validates the provided bundles. The output is the validation report without having to invoke any other endpoint.

The expected input is a .zip file that contains the bundles to validate (there is no need to include all bundles in the inheritance chain if they are available in the operations center). The output is not the effective bundle in .zip format, but the validation report.

An example of the Post request:

curl -s -H 'Accept: application/json' -H 'Content-Type: application/zip;charset=utf-8' --user [USER] --data-binary @[PATH_TO_ZIP] -XPOST $JENKINS_URL/casc-bundle/pre-validate-bundle▼

A report is generated once the bundles are validated. The following is an example of validated bundle report.

{"validations": [
        {
        "bundle": "bundle-2",(1)
        "result":         {
            "valid": true,(2)
            "structureValidations": [],
            "controllerValidations":             [
                                {
                    "controller": "landon-controller",
                    "controllerStatus": "ONLINE",
                    "validations": []
                },
                                {
                    "controller": "controller-2",
                    "controllerStatus": "ONLINE",
                    "validations": []
                }
            ]
        }
    },
        {
        "bundle": "bundle-1",
        "result":         {
            "valid": false,
            "structureValidations": [            {
                "level": "ERROR",(3)
                "validationCode": "BUNDLE_CONTENT",
                "message": "[CONTVAL] - Missing {JOSS} Configuration as Code file specified in the bundle.yaml file."(4)
            }],
            "controllerValidations": []
        }
    }
]}▼

1	"bundle": Bundle name
2	"valid": 'true' indicates the bundle validation was a success. `false` indicates the validation failed.
3	"level": If the bundle validation fails, this field displays with an 'ERROR'.
4	"message": Failed bundle validation error message.

Validating bundles with GitHub checks

Instead of using CLI or HTTP endpoints to validate bundles, you can also use GitHub Checks. When a pull request (PR) is generated against a specific bundle, prevalidation checks are run, and the results are displayed on the Checks tab of the PR.

The GitHub App authentication for your GitHub Organization must be enabled to use the Checks feature.

Figure 9. Bundle validation in the GitHub checks tab

The Checks tab displays prevalidation details for each branch, including error messages.

Configuring the bundle update recurrence period

After the initial configuration on startup, a background process automatically checks for bundle updates every twenty minutes.

Optionally, you can customize this by setting the Java system property com.cloudbees.opscenter.client.casc.ConfigurationUpdaterTask.recurrencePeriod to the desired interval, in minutes.

For example, to check for updates every five minutes, use the following startup command:

java -Dcore.casc.config.bundle=$JENKINS_HOME/casc-bundle-link.yml \
 -Dcom.cloudbees.opscenter.client.casc.ConfigurationUpdaterTask.recurrencePeriod=5 \
 -jar jenkins.war▼

Understanding risks related to updating CasC files

Using CasC gives you great power and flexibility, which introduces the possibility for errors that could impact your installation. Please review the information below to help guide you as you evolve your CasC bundle.

Risks when updating the `bundle.yaml` file

A bundle update is detected only when the version attribute in the bundle.yaml file has been incremented. However, when using the CasC Bundle Retriever for the operations center, you can activate auto-versioning, which replaces the version attribute with the commit hash obtained from your source control management (SCM) system. If the version attribute is not incremented, the bundle update is not detected, and cannot be applied.

A CasC bundle can become invalid if you change the apiVersion:2 to apiVersion:1 while updating it.

Different values for this property result in different behaviors for Plugin management with CasC. CasC bundles with apiVersion:2 automatically manage dependencies and does not require non-CAP plugins in the plugins.yaml file to be also part of the plugin-catalog.yaml file. Therefore, if you update your bundle from apiVersion:2 to apiVersion:1, different validations and behaviors are expected, and the validation process may declare the bundle invalid.

Risks when updating the `plugins.yaml` file

You can install CloudBees Assurance Program (CAP) Tier 1 and Tier 2 plugins by adding the plugins to the plugins.yaml file.

CAP Tier 1 and Tier 2 plugins

To get the list of CloudBees Tier 1 and Tier 2 plugins:

Navigate to CloudBees CI Plugins.
To verify that your plugin is a CAP Tier 1 or Tier 2 plugin, you can either:
- Search for a specific plugin and filter based on Verified or Proprietary (Tier 1) or Compatible (Tier 2).
- Scroll to the bottom of the page and select All Tier 1 plugins to browse all Tier 1 plugins and then filter by Tier.

For more information on the CAP and how CloudBees classifies plugins, refer to CloudBees plugin support policies.

Uninstalling a plugin

There are no "delete/uninstall" operations when using CasC, and you cannot uninstall plugins by removing the plugins from the plugins.yaml file.

To uninstall a plugin:

Uninstall the plugin in the UI.
Before restarting, remove the plugin IDs from the plugins.yaml file. Otherwise, the plugins will be reinstalled.

Risks when updating the `jenkins.yaml` file

This file can be edited to change or add Jenkins configurations. There are a wide variety of possible combinations, depending on the plugins that are configured as code. This file is managed by the Jenkins LTS - Configuration as Code (JCasC) plugin. Therefore, all Jenkins LTS documentation, reported bugs, community blogs, Gitter chats, etc., apply to this file.

The bulk of the content in this file is defined by the various plugins you have installed with only a small portion of it defined by the Jenkins core itself. However, there are a few general items to consider when editing the file:

Removing a configuration from this file does not remove the configuration from an existing controller or operations center instance.
Adding configurations requires the underlying plugin that the configuration references to be installed. Jenkins startup will fail otherwise.
Backward compatibility depends on the specific plugins that are being configured. If a plugin changes its configuration in an incompatible way, then the controller or the operations center will fail to start until the jenkins.yaml file is adapted, or the section in the file that is related to the plugin is removed.
This file may contain data which is tied to a specific CloudBees instance, such as encoded secrets, the operations center connection details, and others, so a jenkins.yaml is, in general, suitable for a single CloudBees controller instance. This might lead to indecipherable secrets, controller to operations center connection failing, and many more things depending on the specific configuration.

Using hot reload on a configuration bundle

CasC allows you to reload a configuration file without restarting the instance. Since the CasC API plugin manages plugin updates, validating a hot reload is a more complex process. A bundle reload is not always possible. In the UI, if the bundle hot reload is not possible, the Reload Configuration button does not appear on the Bundle update tab of the CloudBees Configuration as Code export and update screen.

If plugin upgrades are in the updated bundle, Jenkins must be restarted.

If a hot reload is possible, the following operations are automatically performed:

Install any new plugins.
Call the API on configuration as code to reload the Jenkins configuration.

Configuration bundles that do not include plugin updates can be reloaded without an instance restart. This feature allows you to update Jenkins configurations or install new plugins dynamically.