Troubleshooting CasC for controllers

14 minute readScalabilityAutomation

Validating CasC bundles

Validation of CasC bundles is performed automatically and if problems are detected, a validation message is returned. Each CasC bundle validation message includes the type of validation, a general validation code, and a descriptive message. For example:

ERROR - [VERSIONVAL] - Missing version property in the bundle.yaml file.

There are two types of validation messages: ERROR and WARNING.

  • An ERROR indicates a critical problem with the CasC bundle that must be resolved before the bundle can be applied to the instance.

  • A WARNING indicates the current file is invalid. However, the entire bundle may still be valid.

    For example, when creating items, specific plugins must first be installed. If the bundle contains an items.yaml file with the correct kind property for the item type, but the required plugin is not currently installed, a warning is returned. Since the plugin may be installed once the updated bundle has been applied to the instance, the warning indicates that you should verify the required plugin is included in the plugins.yaml file.

Accessing validation messages

For controller CasC bundles, you can access validation errors and warnings using the following methods:

  • All validation errors and warnings are available in the controller’s log files.

  • When a controller CasC bundle is added to the operations center using the Configuration as Code bundle location and the bundle has structural validation errors or warnings, an administrative monitor is displayed. Select the Configuration as Code bundles link to navigate directly to this screen and view the errors or warnings.

  • When the bundle is assigned to the controller and the bundle has controller validation errors or warnings, the following message is displayed in the operations center dashboard: The bundle has errors or warnings. From the operations center dashboard, select the down arrow to the right of your controller’s name, and then select Manage to view the message details.

General validation codes

The following general validation codes are included in each CasC bundle validation message.

Table 1. General validation codes
CodeDescription

APIVAL

Validates the apiVersion field in the bundle.yaml file.

CATALOGVAL

Validates the plugin-catalog.yaml file in the bundle.

CONTVAL

Validates the content of the bundle.yaml file and that the bundle is properly described.

DESCVAL

Validates the bundle.yaml file.

EMPTY

Validates that the folder containing the bundle is empty.

FORMATVAL

Validates the format of one file in the bundle.

ITEMS

Validates that items can be created.

JCASC

Validates the Jenkins Configuration-as-Code file(s).

LOADERR

Validates that the bundle can be loaded or downloaded before it is applied to the instance. For example, if the operations center is inaccessible and controllers cannot access CasC bundles, LOADERR is returned.

NONEXIST

Validates the existence of the CasC bundle folder or the bundle.yaml file.

PLUGINVAL

Validates that the plugins in the plugins.yaml file can be installed.

RBAC

Validates the Role-Based Access Control (RBAC) configuration.

SEC

Validates that potentially unsecure files are not included in the bundle.

STRUCTURE

Validates the folder containing the bundle in the file system.

UNDEFINED

Undefined validation or unexpected error occurred during a validation. For example, an undefined validation due to legacy code.

VERSIONVAL

Validates the version field in the bundle.yaml file.

Structural validations

The following validations verify the CasC bundle structure, including the files within the bundle and the content of the bundle.yaml file. The validation is performed when the instance is started and the bundle is applied. If an error is returned, the instance does not start and the validation error is included in the log file.

If you add a controller CasC bundle to the operations center using the Configuration as Code bundle location and a problem occurs, an ERROR message is properly returned, but does not prevent the instance from starting or the bundle from being applied to the controller.
Table 2. Structural validations
CodeTypeMessageMeaning

APIVAL

ERROR

Missing apiVersion property in the bundle.yaml file.

The bundle.yaml must contain the apiVersion property.

APIVAL

ERROR

apiVersion property in the bundle.yaml file must be a positive integer.

apiVersion property in the bundle.yaml file must be an integer.

The apiVersion must always be a positive integer, and use a whole number that is greater than zero. -1 and a are examples of invalid values.

CONTVAL

ERROR

Found file-name file specified in the bundle.yaml file that is not a folder or a yaml file.

The file is referenced in the bundle.yaml file, but it is not a folder or a YAML file, so the entry is invalid.

CONTVAL

WARNING

Not referenced file/folder file-name found in bundle folder.

A file has been detected in the bundle, but it is not directly referenced in the bundle.yaml file or the bundle.yaml file does not reference a folder that contains the file.

CONTVAL

ERROR

Missing file-name file specified in the bundle.yaml file.

The file is referenced in the bundle.yaml file, but the file does not exist.

DESCVAL

ERROR

The bundle descriptor is wrong or the bundle.yaml file does not exist.

The bundle.yaml file cannot be parsed. It is either empty, is not a YAML file, or it does not exist.

DESCVAL

ERROR

Missing id property in the bundle.yaml file.

The bundle.yaml file must contain the id property.

EMPTY

ERROR

The CasC bundle folder is empty.

The CasC bundle folder is empty.

NONEXIST

ERROR

The CasC bundle folder does not exist. Impossible to validate.

The folder containing one or more bundles does not exist. This error occurs when the instance is starting and the system property points to a folder that does not exist.

NONEXIST

ERROR

The bundle.yaml file is missing.

All bundles should have a bundle.yaml file that acts as a bundle descriptor. The bundle.yaml file is missing.

PLUGINVAL

WARNING

Plugin plugin-id is not in the CloudBees Assurance Program (CAP) or in applicable plugin catalog(s).

Any plugin in the plugins.yaml file should be either in the CAP or referenced in the plugin-catalog.yaml file. When this WARNING is raised, you must add the plugin-id to the plugin-catalog.yaml` file.

SEC

ERROR

The bundle.yaml file refers an entry that is not a folder or a yaml file file.

The bundle.yaml file references a file entry that is not a folder or a YAML file.

STRUCTURE

WARNING

The following files have been detected in the bundle folder but they are not yaml files: list of files.

Files in the bundle folder are not YAML files. Bundles can only contain YAML files and/or folders containing YAML files.

STRUCTURE

ERROR

The CasC bundle is not a folder. Impossible to validate.

This error occurs when the instance is starting and the system property points to a file instead of a folder.

STRUCTURE

ERROR

Impossible to read folder X content in the bundle: error message

Impossible to read bundle content: error message

An unexpected error occurred when reading the files and folders. Refer to the error message for details.

STRUCTURE

ERROR

The bundle inherits from parent but the parent folder does not exist.

The bundle inherits from a parent bundle, but the parent bundle does not contain a valid bundle.yaml file.

The bundle.yaml file in the child bundle contains the parent property, but the bundle it is referencing does not exist or the parent bundle does not contain a bundle.yaml file.

VERSIONVAL

ERROR

Missing version property in the bundle.yaml file.

The bundle.yaml must contain the version property.

Controller validations

The following validations verify the controller CasC bundles and are performed remotely on the controllers when:

  • A bundle is applied to the controller.

  • The controller is connected to the operations center.

  • A controller is offline and is then placed online.

Table 3. Controller validations
CodeTypeMessageMeaning

CATALOGVAL

ERROR

error-message

The plugin catalog cannot be installed. See the error-message details.

CATALOGVAL

ERROR

The plugin-catalog.yaml file is not a valid yaml/json file.

The plugin-catalog.yaml file has an invalid YAML format and you must review it.

CATALOGVAL

WARNING

More than one plugin catalog file used in bundle_id. Using only first read file.

The bundle descriptor (bundle.yaml file) refers to more than one plugin catalog file and you cannot specify more than one plugin catalog file. Only the file that is read first is applied and the rest of the files are ignored.

ITEMS

ERROR

The bundle.yaml file references list of files item files but they are empty or have an invalid YAML format. Impossible to validate items.

The bundle.yaml file refers to files in the items section that are empty, cannot be found, or do not have a valid YAML format. You should check these files.

ITEMS

ERROR

The bundle.yaml file references list of files RBAC files but they cannot be found. Impossible to validate RBAC configuration for items.

The bundle.yaml file references list of files RBAC files but they are empty or have an invalid YAML format. Impossible to validate RBAC configuration for items.

The bundle.yaml file refers to files in the rbac section that are empty, cannot be found, or do not have a valid YAML format. You should check these files or the RBAC configuration because these items cannot be verified.

ITEMS

WARNING

Impossible to parse file-name or missing removeStrategy attribute. Please review the file format.

The file is not a YAML file or is missing the removeStrategy attribute. You must review the contents of the file.

ITEMS

WARNING

There are unknown attributes in the file-name file: list of unknown attributes.

The file has an invalid format. It should only contain the removeStrategy, root, and items attributes. The unknown attributes must be deleted from the file. For more information, refer to Creating a CasC bundle for controllers.

ITEMS

WARNING

No items in the file-name file. Consider removing it from the CasC bundle.

The file does not contain items or the items.yaml file is empty. This file reference should be removed from the bundle.yaml file.

ITEMS

WARNING

Found items missing the kind in file-name, so it is impossible to create them: list of names.

One or more elements in the items list does not have the mandatory kind attribute that is used to detect the type of item to create or update. You must add or remove the kind attribute.

ITEMS

WARNING

The file-name file cannot be properly parsed:

Impossible to parse item name of type kind. Reason: error-message

Please check the items format. If any property cannot be parsed, verify that the required plugin is included in the plugins.yaml file.

One or more elements in the items list cannot be parsed. For each item with the name and kind properties defined, there is an error message that provides more details. Possible reasons are as follows:

  • One property of the item is misspelled. You must check the file.

  • The instance does not have a required plugin installed, so the symbol cannot be found. You should verify the required plugin is included in the plugins.yaml file.

ITEMS

WARNING

The following types of item in file-name cannot be recognized: list of kind. Please check the items and make sure the needed plugin will be installed through the plugin YAML files.

The kind of one or more elements in the items list cannot be recognized. The possible reasons are as follows:

  • Misspelling of the kind. You should check the file.

  • The instance does not have a plugin, so the symbol cannot be found. You must check the plugins.yaml file.

ITEMS

WARNING

The file-name declares a root directory parent. It cannot be found either in the instance or in the files parsed. Please verify that parent is in a YAML file and that the files are correctly ordered.

The file has the root attribute pointing to the parent, therefore all items are created within the parent folder. However, the parent cannot be found in the instance or it is not defined in any of the already processed items.yaml files. Potential reasons are as follows:

  • The parent item is not defined. You must add a YAML file with that definition.

  • The parent of the item must be defined in a separate YAML file and applied to the instance before it is applied to the file currently being validated. You must check if the items.yaml files are declared in the bundle.yaml file in the correct order.

ITEMS

WARNING

Item name of type kind defines the list of roles roles that cannot be found in any RBAC file.

The item is configuring the RBAC roles but they are not in any rbac.yaml file. Potential situations are as follows:

  • The roles are defined through an external tool such as LDAP.

  • You did not include the RBAC configuration in the bundle and you must check the bundles.yaml and all rbac.yaml files.

JCASC

ERROR

The bundle.yaml file references file in the Jenkins Configuration as Code section that cannot be found. Impossible to validate the Jenkins configuration.

The bundle.yaml file references file in the Jenkins Configuration as Code section that is empty or has an invalid YAML format. Impossible to validate the Jenkins configuration.

The bundle.yaml file references file in the Jenkins Configuration as Code (CasC) for Controllers section that are either empty, cannot be found, or have an invalid YAML format. You must check the jenkins.yaml file.

JCASC

ERROR

It is impossible to validate the Jenkins configuration. Please review your Jenkins and plugin configurations. Reason: error-message

The configuration is invalid and cannot be validated. There may be a misspelling in a symbol or the symbol cannot be configured because a required plugin is not installed. You must verify the plugins in the plugin.yaml file.

JCASC

WARNING

Invalid Jenkins configuration: error-message

The required plugin is installed in the instance, but it cannot be properly configured. The potential reasons are as follows:

  • There is a change in the Jenkins version and the instance must be updated or the configuration file must be downgraded.

  • The plugin must be updated.

  • There is a misspelling in a symbol property.

RBAC

ERROR

RoleMatrixAuthorizationStrategy is not the current authorization strategy or the bundle did not configure it and it is required.

The authorization strategy is not the role-based matrix authorization strategy. This is due to the following:

  • The authorizationStrategy attribute is not configured in the jenkins.yaml file.

  • The authorizationStrategy attribute is not configured in the jenkins.yaml file and the current authorization strategy configured in the UI is not supported by RBAC.

In both cases, the role-based matrix authorization strategy must be added to the jenkins.yaml file to use CloudBees RBAC with CasC. For example, you must add the following to the jenkins.yaml file:

authorizationStrategy: "cloudBeesRoleBasedAccessControl"

RBAC

ERROR

The bundle configures RBAC but the authorization strategy that will be configured is not RoleMatrixAuthorizationStrategy ("CloudBeesRoleBasedAccessControl" in the jcasc section). If you set the authorization strategy in the bundle, please make sure the files are correctly ordered.

The bundle.yaml file refers to several files in the Jenkins Configuration as Code section. Ensure that there is only a single configuration for the authorization strategy.

RBAC

ERROR

The bundle.yaml file refers the {list-of-file-names} JCasC files but they cannot be found. Impossible to validate the needed configuration for RBAC.

The bundle.yaml file refers the {list-of-file-names} JCasC files but they are empty or have an invalid YAML format. Impossible to validate the needed configuration for RBAC.

The bundle.yaml file refers to files in the Jenkins Configuration as Code section that cannot be found, are empty, or do not have a valid YAML format. You must check the files or the RBAC configuration cannot be checked.

RBAC

ERROR

The bundle.yaml file refers the {list-of-file-names} RBAC files but they cannot be found. Impossible to validate RBAC configuration.

The bundle.yaml file refers to the {list-of-file-names} RBAC files but they are empty or have an invalid YAML format. Impossible to validate RBAC configuration.

The bundle.yaml file refers to files in the RBAC section that cannot be found, are empty, or do not have a valid YAML format. You must check the files.

RBAC

ERROR

Impossible to parse file-name. Please review the file format: reason

The file has an incorrect syntax or format and it must be reviewed and updated. reason gives details about the error. The possible reasons are as follows:

  • The removeStrategy property is not included in the file.

  • There are unknown attributes in the rbac.yaml file, followed by the list of unknown attributes.

  • Incorrect syntax

RBAC

ERROR

The following permissions in file-name cannot be recognized: list-of-permissions. Please check the permissions and make sure the needed plugin will be installed through the plugin yaml files.

RBAC needs the permissions configured in the rbac.yaml file to be available to the controller, but the permissions cannot be detected. The possible reasons are as follows:

  • The plugin where the permissions are included is not in the plugins.yaml file.

  • There is a misspelling in the permissions.

RBAC

WARNING

The bundle configures RBAC but the authorization strategy is not configured as RoleMatrixAuthorizationStrategy in the jcasc section. RoleMatrixAuthorizationStrategy is the current authorization strategy, but if this changes, the bundle will become erroneous. It is strongly recommended that you configure the authorization strategy as part of the bundle.

RBAC is currently configured in the Manage Jenkins Configure Global Security page to use the Role-Based Matrix Authorization Strategy and it is not configured in the jenkins.yaml file. If the UI configuration is changed, the CasC bundle will no longer be valid. CloudBees recommends that you override the UI setting by configuring the authorization strategy in the 'jenkins.yaml` file. For example, add the following to the jenkins.yaml file:

authorizationStrategy: "cloudBeesRoleBasedAccessControl"

RBAC

WARNING

No RBAC configuration in the file-name file. Consider removing it from the CasC bundle.

The file does not have roles or groups defined. Remove this file from the CasC bundle.

Exporting the original CasC bundle

You can export the original CasC bundle and compare it to the current configuration to determine if recent changes may have been incorrectly applied.

  1. Sign in as a user with the Administer permission.

  2. Select Manage Jenkins in the left pane.

    CloudBees Configuration as Code export and update
    Figure 1. CloudBees Configuration as Code export and update
  3. Select CloudBees Configuration as Code export and update.

  4. Select Original bundle.

    Original bundle
    Figure 2. Original bundle
  5. Select Download to export the original bundle. You can export individual YAML files or export all files in zip format.

  6. Select Current configuration.

    Current configuration
    Figure 3. Current configuration
  7. Select Download to export the current configuration. You can export individual YAML files or export all files in zip format.

  8. Compare the original bundle to the current configuration to determine if recent changes may have been incorrectly applied.

Validating the bundle deployment in operations center

You can check that the configuration bundle is properly deployed to the operations center server using an HTTP client tool such as cURL. Use the tool to fetch the bundle.yaml using the URL you entered into the casc-bundle-link.yml file. To authenticate the request, you will also need to provide the token value from the casc-bundle-link.yml file as an HTTP header named X-CasC-Token.

Example request with authentication token

curl -v -H "X-CasC-Token: abc123de-f456-7891-gh23-i4jkl45m6n7o" \
http://oc1.example.org:8180/config-bundle/bundle-1

This request should return the bundle.yaml file from the configuration bundle in the directory named bundle-1.

If the bundle.yaml file is not returned and you received the following error: No response/connection refused/timeout.

  • Check that the operations center server is running.

  • Check that the hostname matches the Operation Center’s.

  • Check that the port in the URL matches the operations center’s port.

  • Check whether a firewall between the client and the Operation’s Center could be blocking the connection.

If the bundle.yaml file is not returned and you received the following error: Wrong bundle returned. Check that the bundle files were placed into the correct directory on the operations center server.

If the bundle.yaml file is not returned and you received the following error: HTTP status code 404/Not Found returned.

This can happen when either the URL path was incorrect, or the token was not found.

  • Check that the URL matches the format specified above, and that the final part matches the name of the directory containing the configuration bundle.

  • Check that the token value provided matches the one for this bundle in the bundles list in operations center under Manage Jenkins > Configuration as Code Bundles.

Delaying Configuration as Code (CasC) for Controllers configuration for Jenkins to load global configuration

Jenkins 2.199 introduced a check to prevent saving global configurations before they are loaded. Configuration as Code (CasC) for Controllers needs to apply global configurations before Jenkins loads jobs to correctly reference any global state. Therefore, until JENKINS-51856 is implemented, there is a race condition where Jenkins may fail to start when used with the Configuration as Code plugin.

If you encounter the race condition, Jenkins will fail to start with an exception message similar to the following:

SEVERE	jenkins.InitReactorRunner$1#onTaskFailed: Failed ConfigurationAsCode.init
java.lang.IllegalStateException: An attempt to save the global configuration was made before it was loaded

The workaround for this race condition is to delay Configuration as Code (CasC) for Controllers configuration to give Jenkins time to load the global configuration.

To enable this delay, set the io.jenkins.plugins.casc.ConfigurationAsCode.initialDelay system property to the amount of delay time in milliseconds. This delay time value will be dependent on aspects of your system (cpu/disk) and configuration.

To find a value that will work with your system, start with 5000 (5 seconds) and increment by 2000 (2 seconds) until the issue is corrected, then add 1000 (1 second) to the final value for extra safety.

Enabling Configuration as Code (CasC) for Controllers configuration delay on a client controller

Applying the workaround system property to a client controller depends on the client controller installation method.

Using the same way other system properties have been added to the init script:

In the same way other system properties have been added to the init script, set the -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay system property to the amount of delay time in milliseconds.

Support bundles for CasC

When generating a support bundle, you can include the CloudBees Configuration as Code bundle within the support bundle. Optionally, you can also enable Support Bundle Anonymization from the Configure Global Security screen, to automatically anonymize metadata within the YAML files that are included in the generated support bundle. For more information, see Generating a support bundle.

The CasC API plugin must be installed to include the CasC bundle in the generated support bundle.