Troubleshooting CasC for controllers

Delaying Configuration as Code (CasC) for Controllers configuration for Jenkins to load global configuration

Jenkins 2.199 introduced a check to prevent saving global configurations before they are loaded. Configuration as Code (CasC) for Controllers needs to apply global configurations before Jenkins loads jobs to correctly reference any global state. Therefore, until JENKINS-51856 is implemented, there is a race condition where Jenkins may fail to start when used with the Configuration as Code plugin.

If you encounter the race condition, Jenkins will fail to start with an exception message similar to the following:

SEVERE	jenkins.InitReactorRunner$1#onTaskFailed: Failed ConfigurationAsCode.init
java.lang.IllegalStateException: An attempt to save the global configuration was made before it was loaded

The workaround for this race condition is to delay Configuration as Code (CasC) for Controllers configuration to give Jenkins time to load the global configuration.

To enable this delay, set the io.jenkins.plugins.casc.ConfigurationAsCode.initialDelay system property to the amount of delay time in milliseconds. This delay time value will be dependent on aspects of your system (cpu/disk) and configuration.

To find a value that will work with your system, start with 5000 (5 seconds) and increment by 2000 (2 seconds) until the issue is corrected, then add 1000 (1 second) to the final value for extra safety.

Enabling Configuration as Code (CasC) for Controllers configuration delay on a managed controller/team controller

In the case of a managed controller or team controller, the workaround can be applied as a system property on the configuration page of the controller item in the operations center dashboard as illustrated below:

Managed controllers configuration screen

Figure 1. Managed controller configuration screen

Enabling Configuration as Code (CasC) for Controllers configuration delay on a client controller

Applying the workaround system property to a client controller depends on the client controller installation method.

Using the same way other system properties have been added to the init script:

In the same way other system properties have been added to the init script, set the -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay system property to the amount of delay time in milliseconds.

Exporting the original CasC bundle

You can export the original CasC bundle and compare it to the current configuration to determine if recent changes may have been incorrectly applied.

Sign in as a user with the Administer permission.
Select Manage Jenkins in the left pane.

Figure 2. CloudBees Configuration as Code export and update
Select CloudBees Configuration as Code export and update.
Select Original bundle.

Figure 3. Original bundle
Select Download to export the original bundle. You can export individual YAML files or export all files in zip format.
Select Current configuration.

Figure 4. Current configuration
Select Download to export the current configuration. You can export individual YAML files or export all files in zip format.
Compare the original bundle to the current configuration to determine if recent changes may have been incorrectly applied.

Validating the bundle deployment in operations center

You can check that the configuration bundle is properly deployed to the operations center server using an HTTP client tool such as cURL. Use the tool to fetch the bundle.yaml using the URL you entered into the casc-bundle-link.yml file. To authenticate the request, you will also need to provide the token value from the casc-bundle-link.yml file as an HTTP header named X-CasC-Token.

Example request with authentication token

curl -v -H "X-CasC-Token: abc123de-f456-7891-gh23-i4jkl45m6n7o" \
http://oc1.example.org:8180/config-bundle/bundle-1

This request should return the bundle.yaml file from the configuration bundle in the directory named bundle-1.

If the bundle.yaml file is not returned and you received the following error: No response/connection refused/timeout.

Check that the operations center server is running.
Check that the hostname matches the Operation Center’s.
Check that the port in the URL matches the operations center’s port.
Check whether a firewall between the client and the Operation’s Center could be blocking the connection.

If the bundle.yaml file is not returned and you received the following error: Wrong bundle returned. Check that the bundle files were placed into the correct directory on the operations center server.

If the bundle.yaml file is not returned and you received the following error: HTTP status code 404/Not Found returned.

This can happen when either the URL path was incorrect, or the token was not found.

Check that the URL matches the format specified above, and that the final part matches the name of the directory containing the configuration bundle.
Check that the token value provided matches the one for this bundle in the bundles list in operations center under Manage Jenkins > Configuration as Code Bundles.

Support bundles for CasC

When generating a support bundle, you can include the CloudBees Configuration as Code bundle within the support bundle. Optionally, you can also enable Support Bundle Anonymization from the Configure Global Security screen, to automatically anonymize metadata within the YAML files that are included in the generated support bundle. For more information, see Generating a support bundle.

The CasC API plugin must be installed to include the CasC bundle in the generated support bundle.