Configuration as Code (CasC) for Masters for CloudBees Core on modern cloud platforms

Configuration as Code (CasC) for Masters for CloudBees Core is a Preview feature.
Preview

A Preview feature:

  • Has not undergone end-to-end testing with CloudBees products

  • Is provided without service-level agreements (SLA) and therefore does not include CloudBees' commitment on functionality or performance

  • May impact other stable areas of the product when used

  • May have limited documentation

  • May not be feature complete during the Preview period

  • May graduate from preview state to fully supported or be removed from the product

  • May introduce incompatible, backward-breaking changes that could revoke the ability to upgrade

Product features and documentation are frequently updated. If you find an issue or have a suggestion, please contact CloudBees Support.

Configuration as Code for CloudBees Core is available on the following platforms as a Preview feature:

  • CloudBees Core on traditional platforms 2.204.1.3 (and higher)

  • CloudBees Core on modern cloud platforms 2.204.1.3 (and higher)

Introducing Configuration as Code (CasC) for CloudBees Core

Configuration as Code (CasC) for CloudBees Core simplifies the management of a CloudBees Core cluster by capturing the configuration of CloudBees Core masters in human-readable declarative configuration files which can then be applied to a master in a reproducible way. By capturing the configuration in files, it can be treated as a first class revision-controlled artifact - versioned, tested, validated, and then applied to masters while being centrally managed from CloudBees Core Operations Center.

CasC for CloudBees Core architecture

The process of using CasC for CloudBees Core involves the following steps:

Understanding configuration bundles

The configuration of a master is described in a collection of YAML files referred to as a CasC bundle. The Operations Center can store many different configuration bundles representing a selection of predefined masters. Configuration bundles are stored securely on the Operations Center server which manages the secure distribution of the CasC bundles to the correct masters.

Creating a configuration bundle

To create a new bundle configuration, you must create each of the files listed below. A CasC bundle consists of the following files:

  • bundle.yaml - This file is an index file that describes the bundle, and references the other files in the bundle.

  • jenkins.yaml - This file contains the Jenkins configuration, as defined by the CasC plugin.

  • plugins.yaml - This file contains a list of all the plugins that should be installed on the master.

  • plugin-catalog.yaml - This file contains the plugin catalog definitions that should be created on the master.

    For the bundle to be valid, all of the above files must be present with exactly these filenames.
You will also need a core-casc-security.xml file to configure security. For more information, see Configuring CasC bundle security.

bundle.yaml

The bundle.yaml file is an index file that describes the bundle and references the other files in the bundle. The file must contain all of the properties listed below:

Property Type Description

id

String

This bundle’s unique identifier

version

String

This bundle’s version, this value should be incremented whenever the bundle is modified

description

String

A helpful description of this bundle

plugins/jcasc/catalog

List of Strings

References to the plugins.yaml, jenkins.yaml, and plugin-catalog.yaml files respectively. These sections must appear exactly as specified in the example.

Sample bundle.yaml file

id: "git-bundle"
version: "1"
description: "A simple bundle that sets up a master with the git-client plugin"
plugins:
  - "plugins.yaml"
jcasc:
  - "jenkins.yaml"
catalog:
  - "plugin-catalog.yaml"

jenkins.yaml

This file is the main CasC configuration file that describes the Jenkins master. For more information on how to format this file, see the documentation of the Jenkins Configuration-as-Code plugin.

Sample jenkins.yaml file

jenkins:
  systemMessage: "Jenkins configured using CasC."
  numExecutors: 0
  mode: NORMAL
  securityRealm:
    local:
      allowsSignup: false
      users:
       - id: admin
         password: Qmas9pHXb9wpvp36FaAQ2SBN7XBxsG3h
       - id: newuser
         password: VzHAzoGzHeoq4JF797p8Mb7qL2Ycv54A

plugins.yaml

This file lists all of the plugins that should be installed on the master. All plugins listed must be in the CloudBees Assurance Program (CAP). Plugins outside of CAP (for example, Open Source or your own plugins) should be added via a Plugin Catalog (see the next section).

The contents of this file should be a list of plugin IDs. Find many popular plugins and their IDs at the following sites:

Sample plugins.yaml file

plugins:
  - id: "git"
  - id: "maven-plugin"

plugin-catalog.yaml

This file defines the catalog of plugins that are available for installation on the master. The plugin catalog contains a list of plugins along with the version that should be installed. An optional location can also be specified for plugins that are not available in the standard update centers. For more information on Plugin Catalogs, see Configuring plugin catalogs.

Plugins will not actually be installed on the master unless they are listed in plugins.yaml. Transitive dependencies are not automatically added. See Calculating plugin dependencies for more information on creating a list of transitive dependencies.

Sample plugin-catalog.yaml file

type: plugin-catalog
version: '1'
name: java-catalog
displayName: Java Master Catalog
configurations:
- description: java-builder
  includePlugins:
    artifactory:
      version: 3.3.2
    checkstyle:
      version: 4.0.0
    warnings-ng:
      version: 5.3.0
    my-custom-plugin:
      url: http://www.example.org/jenkins-plugins/my-custom-plugin-1.2.3.hpi

Generating CasC YAML files from pre-existing masters

A blank slate is not ideal when there is a lot to define. It’s easier to configure Jenkins using the UI and then export that configuration as a Configuration Bundle. Configuration as Code (CasC) for CloudBees Core includes a method for creating CasC YAML files from pre-existing masters. However, the generated bundle is not an exact replica of the existing master and needs to be completed manually.

The export feature in Configuration as Code (CasC) for CloudBees Core relies on OSS JCasC export, which has limitations (See JCasC OSS documentation for more information). In other words, the configuration bundle generated by exporting it from an existing master is not offering a complete, correct, and ready-to-use configuration bundle.

One reason the exported bundle is incomplete is that some plugins are only partially compatible with JCasC, which means they can read and load the configuration coming from CasC; however, they are not generating the correct YAML when exporting. Also, the jenkins.yaml section can include extraneous default values for some properties that can be removed.

Exporting a CasC bundle from existing masters involves the following steps:

  1. Create a Test Master in your cluster.

  2. Configure it as you like (plugins and global configuration).

  3. Optionally configure a Plugin Catalog.

  4. Navigate to http://[your_master_url]/core-casc-export.

Prerequisites

To create a CasC bundle from a preexisting master:
  1. Navigate to http://[your_master_url]/core-casc-export. The response will be a composed YAML file containing all the components of the configuration bundle.

    bundle.yaml
    id: "0-client-master"
    description: "Autogenerated bundle descriptor"
    version: "1"
    plugins:
      - "plugins.yaml"
    jcasc:
      - "jenkins.yaml"
    catalog:
      - "plugin-catalog.yaml"
    ---- jenkins.yaml
    jenkins:
      agentProtocols:
      - "Diagnostic-Ping"
      - "JNLP4-connect"
      - "OperationsCenter2"
      - "Ping"
      disableRememberMe: false
      markupFormatter: "plainText"
      mode: NORMAL
      myViewsTabBar: "standard"
      numExecutors: 0
      primaryView:
        all:
          name: "all"
      projectNamingStrategy: "standard"
      quietPeriod: 5
      remotingSecurity:
        enabled: false
      scmCheckoutRetryCount: 0
      slaveAgentPort: 0
      systemMessage: |+
        v3 - Master configured automatically by Core CasC from CJOC
    
      updateCenter:
        sites:
        - id: "virtual-cap-core-cm"
          url: "https://jenkins-updates.cloudbees.com/update-center/envelope-core-cm/update-center.json"
      views:
      - all:
          name: "all"
      viewsTabBar: "standard"
    security:
      apiToken:
        creationOfLegacyTokenEnabled: false
        tokenGenerationOnCreationEnabled: false
        usageStatisticsEnabled: true
      sSHD:
        port: -1
    unclassified:
      contentFilters:
        enabled: false
      experimentalPlugins:
        enabled: false
      location:
        adminAddress: "address not configured yet <nobody@nowhere>"
        url: "http://127.0.0.1.beesdns.com:8080/"
      mailer:
        charset: "UTF-8"
        useSsl: false
      operationsCenterRootAction:
        connectionDetails: "----- BEGIN CONNECTION DETAILS -----\nH4sIAAAAAAAAAA3KQQ6DIBBA0bvMugNDkeBwG8TRmihtYLpqenfJ3738H+wtV4UEW8jBcWD0PjJO\r\
          \nvBEumQllKsHNJc7REzzgWMdLWM5DquKVu0ob/G3n8JfqJ1nrntHQyJlFpK+1m/K+EhOThf8NZy3p\r\
          \n8XIAAAA=\r\n----- END CONNECTION DETAILS -----\n"
      pollSCM:
        pollingThreadCount: 10
    plugins.yaml
    plugins:
    - id: "trilead-api"
    - id: "configuration-as-code"
    - id: "mapdb-api"
    - id: "junit"
    - id: "cloudbees-folder"
    - id: "display-url-api"
    - id: "cloudbees-uc-data-api"
    - id: "operations-center-client"
    - id: "chucknorris"
    - id: "cloudbees-analytics"
    - id: "command-launcher"
    - id: "metrics"
    - id: "mailer"
    - id: "nectar-rbac"
    - id: "operations-center-context"
    - id: "async-http-client"
    - id: "script-security"
    - id: "cloudbees-folders-plus"
    - id: "variant"
    - id: "blueocean-commons"
    - id: "jaxb"
    - id: "jdk-tool"
    - id: "cloudbees-license"
    - id: "structs"
    - id: "support-core"
    - id: "scm-api"
    - id: "token-macro"
    - id: "nectar-license"
    - id: "operations-center-agent"
    - id: "cloudbees-administrative-monitors"
    - id: "cloudbees-assurance"
    - id: "credentials"
    - id: "workflow-api"
    - id: "jackson2-api"
    - id: "cloudbees-blueocean-default-theme"
    - id: "workflow-step-api"
    - id: "bouncycastle-api"
    - id: "apache-httpcomponents-client-4-api"
    - id: "javadoc"
    - id: "ldap"
    - id: "pam-auth"
    - id: "matrix-auth"
    - id: "windows-slaves"
    - id: "antisamy-markup-formatter"
    - id: "matrix-project"
    - id: "beer"
    plugin-catalog.yaml
    type: "plugin-catalog"
    version: "1"
    name: "must-have-catalog"
    displayName: "Must Have Master Catalog"
    configurations:
    - description: "Important plugins"
      includePlugins:
        beer:
          url: "http://jenkins-updates.cloudbees.com/download/plugins/beer/1.2/beer.hpi"
        chucknorris:
          url: "http://jenkins-updates.cloudbees.com/download/plugins/chucknorris/1.1/chucknorris.hpi"
  2. The page is divided into different sections per CasC file in the bundle. Copy the sections into separate YAML files named as noted below and put them into a folder:

    • bundle.yaml

    • jenkins.yaml

    • plugins.yaml

    • plugin-catalog.yaml

      For the bundle to be valid, all of the above files must be present with exactly these filenames.

Adding a CasC bundle to Operations Center

Configuration bundles are made available from the Operations Center to connected masters using HTTP/HTTPS. Configuration bundles are stored on the Operations Center server as a directory containing the bundle files. Multiple CasC bundles can be deployed to the Operations Center server and are all stored in $JENKINS_HOME/jcasc-bundles-store.

Example directory for multiple CasC bundles

$JENKINS_HOME/jcasc-bundles-store/
├── bundle-1
│   ├── bundle.yaml
│   ├── jenkins.yaml
│   ├── plugin-catalog.yaml
│   └── plugins.yaml
└── bundle-2
    ├── bundle.yaml
    ├── jenkins.yaml
    ├── plugin-catalog.yaml
    └── plugins.yaml

Manipulating Core CasC bundles using kubectl

Configuration bundles are stored on the Operations Center in the same way as on traditional platforms, but some additional steps are required to manipulate the files. The following sections explain how to manipulate the bundles using kubectl, the Kubernetes command-line tool. The following sample text will be used where you need to substitute your own values:

  • <NAMESPACE> - the Kubernetes namespace into which you installed CloudBees Core for modern cloud platforms

  • <BUNDLE_DIR> - the local directory containing your bundle files

Users loading bundles from an SCM do not need to use kubectl to manage bundles. For more information, see Loading CasC bundles from an SCM.

Listing the bundles currently deployed to the Operations Center

Run the following command to display a list of the bundles that are currently deployed to the Operations Center and available for use:

kubectl exec --namespace <NAMESPACE> cjoc-0 -- \
ls $JENKINS_HOME/jcasc-bundles-store/ | grep -v core-casc-security.xml

Example command to list the bundles deployed to the Operations Center running in namespace cloudbees-core:

kubectl exec --namespace cloudbees-core cjoc-0 -- \
ls $JENKINS_HOME/jcasc-bundles-store/ | grep -v core-casc-security.xml

Example output:

bundle1
bundle2
bundle3

Deploying a bundle to the Operations Center

Copy the directory containing the bundle files to the Operations Center using the following command:

kubectl cp --namespace <NAMESPACE> <BUNDLE_DIR> \
cjoc-0:$JENKINS_HOME/jcasc-bundles-store/

Example command to deploy the bundle directory ‘my-new-bundle’ to the Operations Center running in namespace cloudbees-core:

kubectl cp --namespace cloudbees-core my-new-bundle \
cjoc-0:$JENKINS_HOME/jcasc-bundles-store/

Removing a bundle from the Operations Center

Delete the bundle directory on the Operations Center using the following command:

kubectl exec --namespace <NAMESPACE> cjoc-0 -- \
rm -rf $JENKINS_HOME/jcasc-bundles-store/<BUNDLE_DIR>

Example command to delete the bundle named ‘my-old-bundle’ from the Operations Center running in namespace cloudbees-core:

kubectl exec --namespace cloudbees-core cjoc-0 -- \
rm -rf $JENKINS_HOME/jcasc-bundles-store/my-old-bundle

Updating a bundle on the Operations Center

To update a bundle on the Operations Center, first remove the bundle and then copy and deploy the updated bundle.

Loading CasC bundles from an SCM

Operations Center distributes CasC bundles to masters from local storage; however, CasC bundles can also be pulled from an external source control management (SCM) system using a Freestyle job in Operations Center that can be triggered manually or using a webhook.

Prerequisites

CasC bundles can be in one or more SCM repositories; however, this guide assumes all bundles are in the same Git repository, with the following layout:

mycompany.com/config-bundles.git
- my-master-1
  -- bundle.yaml
  -- jenkins.yaml
  -- plugins.yaml
  -- plugin-catalog.yaml
- my-master-2
  -- bundle.yaml
  …

Creating a job in Operations Center

Operations Center distributes bundles from local storage, so to use files from an external SCM system you must create a Freestyle job to pull those bundles from the SCM to local storage.

To create a job to pull CasC bundles from an SCM to local storage:

  1. Create a Freestyle job in Operations Center.

    You must use a Freestyle job as Pipeline jobs are not supported in Operations Center.
  2. In the Source Code Management section, select an SCM that is supported by Jenkins and enter the repository location.

  3. In the Build section, add an Execute Shell step with the following script:

    cp -r my-master-1 $JENKINS_HOME/jcasc-bundles-store
    cp -r my-master-2 $JENKINS_HOME/jcasc-bundles-store
    $JENKINS_HOME/ is the default Jenkins home directory in CloudBees Core on modern cloud platforms. Update the script if you are running some other CloudBees Core instance.
  4. Save and trigger a build.

Afterwards, CasC bundles should be available in local storage and ready to be used by masters following the normal procedure.

Triggering the job automatically using Webhooks

One way to trigger a Freestyle job to pull CasC bundles from an SCM is through the use of webhooks. If your SCM supports Webhooks, configure a webhook in the SCM to trigger builds on the importing job on push events.

Setting up a webhook through your SCM will ensure a change made in the repository will be automatically available to masters using the CasC bundle.

Configuring CasC bundle security

Core CasC bundles may contain sensitive information, so access to them is protected using a secret token. Access to the bundles is controlled via a configuration file in the Core CasC bundle store:

$JENKINS_HOME/
└── core-casc-security.xml

This file lists all the bundles along with the secret access token required to grant access to it. In this way, you can ensure that a master cannot fetch a bundle that is not intended for it, as well as preventing access by clients posing as masters. If a bundle file is not listed in the core-casc-security.xml file, then it will not be accessible at all.

Sample core-casc-security.xml file

<com.cloudbees.opscenter.server.casc.BundleStorage_-AccessControl>
    <access>
        <entry>
            <string>bundle-1</string>
            <hudson.util.Secret>abc123def456</hudson.util.Secret>
        </entry>
        <entry>
            <string>bundle-2</string>
            <hudson.util.Secret>zyx987wvu654</hudson.util.Secret>
        </entry>
    </access>
</com.cloudbees.opscenter.server.casc.BundleStorage_-AccessControl>

This file describes two bundles:

  • bundle-1 with access token abc123def456

  • bundle-2 with access token zyx987wvu654

When adding a new bundle, create a new <entry/> inside <access>:

Attribute Description

<string>

The name of the directory containing the bundle

<hudson.util.Secret>

The access token value

For example, to add a new bundle, bundle-3, with an access token of d4b0cf30-4b3a-480f-8464, add the following <entry> to the XML above:

<entry>
  <string>bundle-3</string>
  <hudson.util.Secret>d4b0cf30-4b3a-480f-8464</hudson.util.Secret>
</entry>

The access token value is sensitive and should only be shared with masters for which this bundle is intended; the value can be stored encrypted in the core-casc-security.xml. For more information on additional protection while at rest, see Advanced topics.

The core-casc-security.xml file will be re-read when modified and can be updated without restarting the Operations Center.

Updating the core-casc-security.xml file using kubectl

In the commands below, <NAMESPACE> is the Kubernetes namespace into which you installed CloudBees Core for modern cloud platforms.

To update the core-casc-security.xml file on Core for modern cloud platforms:

  1. Download the core-casc-security.xml file using the following command:

    kubectl cp --namespace <NAMESPACE> \
    cjoc-0:$JENKINS_HOME/core-casc-security.xml core-casc-security.xml

    Example command using namespace cloudbees-core:

    kubectl cp --namespace cloudbees-core \
    cjoc-0:$JENKINS_HOME/core-casc-security.xml core-casc-security.xml
  2. Edit the core-casc-security.xml file locally.

  3. Upload the modified core-casc-security.xml file using the following command:

    kubectl cp --namespace <NAMESPACE> core-casc-security.xml \
    cjoc-0:$JENKINS_HOME/core-casc-security.xml

    Example command using namespace cloudbees-core:

    kubectl cp --namespace cloudbees-core core-casc-security.xml \
    cjoc-0:$JENKINS_HOME/core-casc-security.xml

Updating a CasC bundle

A configuration bundle can be updated on the Operations Center and will then be automatically distributed to the Masters that were initialized with that bundle.

After the initial configuration on startup, there is a background process in the master which is checking for updates of the configuration bundle on Operations Center. If there is an update, then the update process shows an Administrative Monitor in the master to inform the user about the new available configuration.

A bundle update will only be detected when the version attribute specified in the bundle.yaml file has changed from its previous value.

When a Master finds an update to its configuration bundle, an alert is shown in the ‘Manage Jenkins’ area.

To apply the update when the Master must be restarted; select an appropriate time to do this and then click Safe Restart to have the Master restart and apply the updated configuration bundle.

In some instances, a restart will not be required to apply an update to the the Master. See Using hot reload on a configuration bundle for more information.

By default Masters will check for updates every twenty minutes, but this can be customized by setting the Java system property com.cloudbees.opscenter.client.casc.ConfigurationUpdaterTask.recurrencePeriod to the desired interval, in minutes. For example, to have a Client Master check for updates every five minutes, use the following startup command:

java -Dcore.casc.config.bundle=$JENKINS_HOME/core-casc-bundle-link.yaml \
 -Dcom.cloudbees.opscenter.client.casc.ConfigurationUpdaterTask.recurrencePeriod=5 \
 -jar jenkins.war
updating bundle

When initializing a Managed Master using CloudBees Core on modern cloud platforms, system properties can be added in the Provisioning options when the Master is being created. You can add the same recurrencePeriod property to the Java Options section in the UI. For example, to check for updates every 5 minutes add the Java Option highlighted in red in the screenshot below:

Adding recurrencePeriod property to Java Options in UI

Using hot reload on a configuration bundle

CloudBees Core Configuration as Code (CB Core CasC) allows users to reload a configuration file without restarting the instance. Since the CloudBees Core CasC plugin manages plugin updates and the Plugin Catalog, validating a hot reload is a more complex process. A bundle reload is not always possible. If there are plugin upgrades in the incoming bundle, then Jenkins needs to be restarted.

If the reload is possible, then the following operations are automatically performed:

  • Update the Plugin Catalog

  • Install any new plugins

  • Call the API on configuration-as-code to reload the Jenkins configuration

Configuration bundles that do not include plugin update requirements can be reloaded without an instance restart. This feature allows users to update Jenkins configurations, install new Plugin Catalogs, or install new plugins dynamically.

Setting up a Managed Master

Masters send requests for their bundles through a secure HTTP(S) endpoint in Operations Center. Each master is aware of which bundle it is connected to through the bundle-link.yaml file that contains the URL of the bundle to use and the access token.

When using CloudBees Core on modern cloud platforms, it is not necessary to create the bundle link file and add the startup options for the Managed Master; these steps are taken care of automatically. To apply a CloudBees Core CasC bundle to a master on CloudBees Core on modern cloud platforms, all you need to do is name the master the same name as the bundle. When creating a Managed Master, Operations Center will look for a CasC bundle with the same name/ID. If one is found, then the Managed Master will automatically be initialized with that bundle.

When using Operations Center to provision a new Managed Master (or Team Master), if the bundles’ storage does not contain a bundle with a name that matches the master name, a link file is automatically created and connected to the master.

managed master casc
To set up a Managed Master on CloudBees Core on modern cloud platforms:
The following steps assume you have already created and deployed a bundle named java-master.
Each configuration bundle is protected by an access token. Before connecting a master to a bundle, a token needs to be created. This process is manual (at least during the Preview Release). See Configuring CasC bundle security for more information.
  1. Navigate to the Operations Center in your web browser.

  2. Select New Item.

  3. Enter ‘java-master’ as the Name.

    The name entered here must be identical to the name of the bundle that was uploaded.
  4. Select Managed Master as the Type.

  5. Click OK.

  6. Customize the Managed Master as normal.

    You do not need to select any additional options here to initialize the master with the java-master CasC bundle.
  7. Click Save. The Managed Master will start and be initialized with the CasC bundle ‘java-master’.

Setting up a Team Master

Configuration bundles can be applied to Team Master. The association mechanism used for Team Master is the same mechanism used for Managed Master. The Team name must match the new CasC bundle name.

Plugins and Plugin Catalogs coming from creation recipes, which are unique to Team Master, are ignored when using a CasC bundle for a Team Master. The CasC bundle takes precedence over the creation recipe values (even though other parameters in recipes are still applied, like provisioning configuration).

Advanced topics

Enabling HTTPS

The security token and CasC bundle both contain sensitive information, so it is recommended that they are protected by using HTTPS when sent between the Operations Center and master. HTTPS must be enabled on the Operations Center side. Typically enabling HTTPS is done by terminating the secure connection externally in a reverse proxy, but it can also be terminated in the Jenkins server. To enable HTTPS, add startup parameters to specify the HTTPS port, server certificate, and key.

Example startup parameters

java -jar jenkins.war --httpsPort=8443 \
--httpsCertificate=/etc/certs/jenkins-cert.cert \
--httpsPrivateKey=/etc/certs/jenkins-cert.key

Using a self-signed certificate

This configuration is not recommended and should not be used in production.

If you are using this method in a limited scope for test deployments, see “Set-up SSL on a CJP environment with a self-sign SSL certificate on each Jenkins box”, a CloudBees knowledgebase article, for guidance on setting up CloudBees Core with self-signed certificates.

Encrypting tokens in core-casc-security.xml

The core-casc-security.xml file contains tokens which can be used to access CasC bundles. These tokens can be stored in the file in an encrypted form to protect them while at rest.

To encrypt a token value, you will need to use the Groovy script console:

  1. Open the UI of the Operations Center server.

  2. Navigate to Manage Jenkins > Script Console.

  3. Enter the following Groovy into the script console:

    String token = "<token>";
    println(new hudson.util.Secret(token).getEncryptedValue());
  4. Replace the text <token> for your token value.

  5. Click Run.

  6. Copy the value under Result.

  7. Replace the unencrypted value in core-casc-security.xml with the copied value. For example:

    <entry>
      <string>bundle-1</string>
      <hudson.util.Secret>abc123def456</hudson.util.Secret>
    </entry>

    becomes:

    <entry>
      <string>bundle-1</string>
      <hudson.util.Secret>{AQAAABAAAAAQz3VMsDr77VJ4aaiVsy/HXYCZogqcthFiUAu6kGOtico=}</hudson.util.Secret>
    </entry>

Calculating plugin dependencies

The Plugin Installation Manager Tool can be used to assist with determining the full list of a plugin’s transitive dependencies. The tool takes as input a plugins.yaml that lists plugins you would like to install and outputs a list of all the plugins to be installed, along with all their transitive dependencies. This list can then be used to populate the plugin catalog so the plugins specified in plugins.yaml can be successfully installed.

The plugins.yaml file used by the tool is not the same as the one created previously when authoring the CasC bundle.

Prerequisites

  • The plugin.yaml file from the CasC bundle

  • Java Developer Kit (JDK) installed and configured

  • Apache Maven installed and configured

  • (optionally) git installed and configured

  • Access to the jenkins.war file that will be used to create the connected Master

To create a list of plugins to be installed including all transitive dependencies:

  1. Clone (or download) the Plugin Installation Manager Tool’s repository from https://github.com/jenkinsci/plugin-installation-manager-tool.

  2. Build the tool using the following commands:

    cd plugin-installation-manager-tool
    mvn clean package
  3. Using the plugins.yaml file from the CasC bundle, create a plugins.yaml file for use with the tool. Rename the CasC bundle’s plugins.yaml file’s YAML property id to artifactId as illustrated below:

    plugins.yaml from CasC bundle plugins.yaml for use with tool
    plugins:
      - id: "git"
      - id: "maven-plugin"
    plugins:
      - artifactId: "git"
      - artifactId: "maven-plugin"
  4. Run the tool, using the following command, updating it with the location of your master .war file and the location of your plugins.yaml file:

    cd plugin-installation-manager-tool
    java -jar \
    plugin-management-cli/target/jenkins-plugin-manager-*.jar \
    --no-download --plugin-download-directory plugin-download \
    --list \
    --war $JENKINS_HOME/cloudbees-core-cm.war \  // (1)
    --plugin-file plugins.yaml // (2)
    1 location of master .war file
    2 location of plugins.yaml file

Sample output file with list of plugins

File containing list of plugins to be downloaded: plugins.yaml
Reading in plugins from plugins.yaml

Plugin download location: plugin-download
No CLI option or environment variable set for update center, using default of https://updates.jenkins.io
No CLI option or environment variable set for experimental update center, using default of https://updates.jenkins.io/experimental
No CLI option or environment variable set for incrementals mirror, using default of https://repo.jenkins-ci.org/incrementals
Will use war file: $JENKINS_HOME/jenkins.war

Installed plugins:

Bundled plugins:

Set of all requested plugins:
chucknorris 1.2
workflow-step-api 1.14

Set of all requested plugins that will be downloaded:
chucknorris 1.2
workflow-step-api 1.14

Set of all existing plugins and plugins that will be downloaded:
chucknorris 1.2
workflow-step-api 1.14
Done

Find the section in your output titled Set of all requested plugins that will be downloaded. This section is the list of plugins that will need to be added to plugin-catalog.yaml to successfully install the plugins.yaml.

You may get significantly more output than the example, especially in the “Bundled Plugins” and “Set of all existing plugins and plugins that will be downloaded” sections.

Troubleshooting

Delaying CasC configuration for Jenkins to load global configuration

Jenkins 2.199 introduced a check to prevent saving global configurations before they are loaded. Configuration as Code (CasC) for Masters for CloudBees Core (CasC) needs to apply global configurations before Jenkins loads jobs to correctly reference any global state. Therefore, until JENKINS-51856 is implemented, there is a race condition where Jenkins may fail to start when used with the Jenkins Configuration as Code (JCasC) plugin.

If you encounter the race condition, Jenkins will fail to start with an exception message similar to the following:

SEVERE	jenkins.InitReactorRunner$1#onTaskFailed: Failed ConfigurationAsCode.init
java.lang.IllegalStateException: An attempt to save the global configuration was made before it was loaded

The workaround for this race condition is to delay CasC configuration to give Jenkins time to load the global configuration.

To enable this delay, set the io.jenkins.plugins.casc.ConfigurationAsCode.initialDelay system property to the amount of delay time in milliseconds. This delay time value will be dependent on aspects of your system (cpu/disk) and configuration.

To find a value that will work with your system, start with 5000 (5 seconds) and increment by 2000 (2 seconds) until the issue is corrected, then add 1000 (1 second) to the final value for extra safety.

Enabling CasC configuration delay on a Client Master

Applying the workaround system property to a Client Master depends on the Client Master installation method.

Using the same way other system properties have been added to the init script:

In the same way other system properties have been added to the init script, set the -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay system property to the amount of delay time in milliseconds.

Enabling CasC configuration delay on a Managed / Team Master

In the case of a Managed or Team Master, the workaround can be applied as a system property on the configuration page of the Master item in the Operations Center dashboard as illustrated below:

Managed Masters configuration page

Validating the bundle deployment in Operations Center

You can check that the configuration bundle is properly deployed to the Operations Center server using an HTTP client tool such as cURL. Use the tool to fetch the bundle.yaml using the URL you entered into the bundle link file. To authenticate the request, you will also need to provide the token value from the bundle link file as an HTTP header named X-CasC-Token.

Example request with authentication token

curl -v -H "X-CasC-Token: abc123def456" \
http://oc1.example.org:8180/config-bundle/bundle-1

This request should return the bundle.yaml file from the configuration bundle in the directory named bundle-1.

If the bundle.yaml file is not returned and you received the following error: No response/connection refused/timeout.

  • Check that the Operations Center server is running.

  • Check that the hostname matches the Operation Center’s.

  • Check that the port in the URL matches the Operations Center’s port.

  • Check whether a firewall between the client and the Operation’s Center could be blocking the connection.

If the bundle.yaml file is not returned and you received the following error: Wrong bundle returned. Check that the bundle files were placed into the correct directory on the Operations Center server.

If the bundle.yaml file is not returned and you received the following error: HTTP status code 404/Not Found returned.

This can happen when either the URL path was incorrect, or the token was not found.

  • Check that the URL matches the format specified above, and that the final part matches the name of the directory containing the configuration bundle.

  • Check that the token value provided matches the one for this bundle that was specified in the core-casc-security.xml file. If the token value in the core-casc-security.xml file has been encrypted, try re-encrypting the token value and checking/correcting the value in the core-casc-security.xml file.