Installing non-CAP plugins with plugin catalogs

The Beekeeper Upgrade Assistant feature of your Operations Center’s Manage Jenkins area is the main interface and entry point to the CloudBees Assurance Program.

Beekeeper Upgrade Assistant manages appropriate upgrades (and downgrades) of plugins on your Operations Center instance, in accordance with the CloudBees Assurance Program.

On Client Masters, the Beekeeper Upgrade Assistant feature is also available, and:

  • Is accessible through a Client Master’s Manage Jenkins area

  • Performs exactly the same types of management activities (as this feature does for your Operations Center instance) on their respective Client Master instances.

However, if you also need to install non-CloudBees Assurance Program plugins, such as plugins which are unknown to CloudBees (perhaps developed internally within your organization) or older versions of plugins, you can configure a plugin catalog. A plugin catalog is used by the Beekeeper Upgrade Assistant to widen the acceptable scope of plugins beyond those defined by the CloudBees Assurance Program. This scope includes the ability to specify plugin dependency rules and acceptable version ranges for these plugins. Your plugin catalog can also specify a Maven 2 artifact repository from which to download plugins.

Once a plugin catalog is defined and is added to your Operations Center instance, it can then be deployed to your connected Client Master instances.

Defining a plugin catalog

Before you can add a plugin catalog to your Operations Center instance, you must first define it.

A plugin catalog is defined in JSON file format, which in turn defines one or more plugin configurations. A plugin configuration defines the set of rules for one or more plugins (some potentially dependencies) and their versions, which can be installed to your Operations Center instance, along with the CloudBees Assurance Program-defined plugins.

To create this file, using your favorite text editor, copy and paste the unannotated empty template below into a new file and save it with an appropriate file name, similar to that of the name or displayName value.

The following JSON content is an example of the contents of a plugin catalog file, whose annotations are explained underneath. An appropriate name for this plugin catalog file might be
additional-pipeline-plugins-catalog.json. Use this information to help complete your own plugin catalog file.

Annotated plugin catalog file

{
    "type" : "plugin-catalog", (1)
    "version" : "1", (2)
    "name" : "additional-pipeline-plugins", (3)
    "displayName" : "Additional Pipeline plugins", (4)
    "configurations" : [ (5)
        {
            "description" : "Additional Pipeline plugins for masters versions 2.61 and 2.73", (6)
            "prerequisites": { (7)
                "productVersion": "[2.60, 2.73]"
            },
            "includePlugins" : { (8)
                "tier3-plugin": {
                    "groupId": "com.my-company.jenkins.plugins", (9)
                    "version" : "1.8.2"
                },
                "my-corporate-plugin": {
                    "url" : "https://download.my-company.com/jenkins/my-corporate-plugin-3.2.1.hpi"
                },
                "my-other-corporate-plugin": {
                    "sha1" : "SC/TzSN+eOrewaDZJZyzLpIQV7E=", (10)
                    "url" : "https://download.my-company.com/jenkins/my-other-corporate-plugin-1.2.3.hpi" (11)
                }
            }
        },
        { (12)
            "description" : "Additional Pipeline plugins for masters versions 2.89+",
            "prerequisites": {
                "productVersion": "[2.89)"
            },
            "includePlugins" : {
                "tier3-plugin": {
                    "groupId": "com.my-company.jenkins.plugins",
                    "version" : "1.8.3"
                },
                "my-corporate-plugin": {
                    "url" : "https://download.my-company.com/jenkins/my-corporate-plugin-3.4.0.hpi"
                },
                "my-other-corporate-plugin": {
                    "sha1" : "SC/TzSN+eOrewaDZJZyzLpIQV7E=",
                    "url" : "https://download.my-company.com/jenkins/my-other-corporate-plugin-1.3.0.hpi"
                }
            }
        }
    ],
    "settings": { (13)
        "httpCredentialsProvider": { (14)
            "credentials": [ (15)
                {
                    "authenticationScope": "ANY", (16)
                    "id": "download-server-credentials" (17)
                }
            ]
        },
        "repository": { (18)
            "layout": "maven2", (19)
            "url": "https://repo.my-company.com/content/repositories/dev-connect"
        }
    }
}
1 type (required) - must have the fixed value plugin-catalog, which defines this JSON file as a plugin catalog file.
2 version (required) - must currently have the value 1, which defines the current metadata format for the plugin catalog.
3 name (required) - the internal name used to identify the plugin catalog. Avoid using spaces in this name.
4 displayName (optional) - a human-readable name for the plugin catalog. This is the name of the plugin catalog that appears in the Operations Center interface.
5 configurations (required) - the plugin catalog can define one or more plugin configurations within this array, where each plugin configuration is represented by an individual element of this array. At least one plugin configuration is required.
6 description - provides more context regarding the configuration itself.
7 prerequisites (optional) - an object which allows the definition of the productVersion member, which defines a range of supported Client Master versions (using the Maven Version Range Specification) for the plugin configuration. If the prerequisites object (and hence its productVersion member) is omitted, then any Client Master version is supported.
8 includePlugins (optional) - an object which allows you to specify the ID values of one or more plugins to include as part of this plugin configuration. For each plugin (specified by its plugin-id), specify the pair made up of the supported groupId and version for the plugin or the URL from where download it.
9 groupId (optional) - provides the groupId for the plugin when its download will take place from a maven repository. If the groupId is not provided, this field will consider org.jenkins-ci.plugins as default value.
10 sha1 (optional) - the SHA1 hash of the plugin that will be used to verify the downloaded plugin.
11 url (optional) - the download URL for the plugin.
12 (Optional) - an additional plugin configuration that provides support for different plugin versions (or plugins themselves), for different Client Master versions (specified in the prerequisites > productVersion member).
13 settings (optional) - the plugin catalog defines all parameters used in the download of the plugins.
14 httpCredentialsProvider (optional) - the plugin catalog defines a set of credentials using the interface CredentialsProvider.
15 credentials (required) - the array containing the credentials. Currently must contain only one element.
16 authenticationScope (required) - if the credentials are defined, must currently have the value ANY.
17 id (required) - the ID of a credential that must be defined in the Client Master where the plugin catalog will be installed. This credential will be used for plugin downloads from servers which require authentication.
18 repository (optional) - an object which allows the definition of a Maven 2 repository from which the plugins (defined in the configurations array above) can be downloaded. If a plugin’s url has not been specified within its relevant configurations section, then the Client Master will attempt to download the plugin from this repository‍'s url. Defining a repository is useful when you need to restrict a Client Master’s access to the Internet for security reasons (part of a process known as air gapping), [1] but want the Client Master to install and update plugins from the repository, which can be achieved by setting up a proxy artifact repository.
19 layout (optional) - must currently have the value maven2. However, if this value is not defined, it is assumed to be maven2 by default.

Copyable plugin catalog file

{
    "type" : "plugin-catalog",
    "version" : "1",
    "name" : "",
    "displayName" : "",
    "configurations" : [
        {
            "description" : "",
            "prerequisites": {
                "productVersion": ""
            },
            "includePlugins" : {
                "replace-with-appropriate-plugin-id": {
                    "groupId": "",
                    "version" : ""
                },
                "replace-with-other-appropriate-plugin-id": {
                    "sha1" : "",
                    "url" : ""
                }
            }
        }
    ],
    "settings": {
        "httpCredentialsProvider": {
            "credentials": [
                {
                    "authenticationScope": "ANY",
                    "id": ""
                }
            ]
        },
        "repository": {
            "layout": "maven2",
            "url": ""
        }
    }
}

Adding a plugin catalog

Once you have defined your plugin catalog, you can add it to your Operations Center instance using the Jenkins CLI tool. This section assumes you have configured an alias for the Jenkins CLI tool.

See the CLI user guide for more information on downloading and configuring the Jenkins CLI tool.
  1. In a terminal/command prompt (window), change to the directory containing the plugin catalog file you defined in Defining a plugin catalog.

  2. Enter the command:

    jenkins-cli plugin-catalog --put < plugin-catalog-file.json

    where plugin-catalog-file.json is the name of your plugin catalog file.
    For example, if you used the plugin catalog example above and saved it to a file named
    additional-pipeline-plugins-catalog.json, then you would enter the command:

    jenkins-cli plugin-catalog --put < additional-pipeline-plugins-catalog.json

    If the command was successful, the Jenkins CLI tool returns the following JSON response:

    {
      "id": "additional-pipeline-plugins",
      "message": "Catalog updated correctly",
      "status": "SUCCESS"
    }

    You can now lvalidate the plugin catalog’s suitability for deployment on a configured Client Master.

Note:

  • If the command was not successful, for example - the JSON format of the plugin file was malformed, the Jenkins CLI tool returns an error like:

    {
      "message": "Unexpected character ('}' ... [Source: ... line: 16, column: 14]",
      "status": "FAILURE"
    }

Updating a plugin catalog on Operations Center

If one or more plugins (defined in a plugin catalog) have updates, you may need to specify these plugin updates in the plugin catalog and then update this plugin catalog on Operations Center.

This process is identical to adding the plugin catalog to Operations Center:

  1. Modify the plugin catalog file defined previously (for example, with the updated plugin versions).

  2. Re-add the plugin catalog to Operations Center.

You should then re-validate and then re-deploy the updated plugin catalog to the Client Masters on which the older version of this plugin catalog had been deployed.

If you are unsure of which Client Masters the older version of the plugin catalog had been deployed to, verify the plugin catalog’s deployment on the relevant Client Masters.

Validating a plugin catalog’s suitability for a Client Master

Once you have added a plugin catalog to Operations Center, you can verify this on the Operations Center interface when validating the plugin catalog’s suitability for deployment to a configured Client Master.

This validation process verifies:

  • That the Client Master version and plugin configuration satisfies exactly one plugin catalog definition

  • That plugin dependencies are resolved

  • That credentials referenced in the plugin catalog are defined in the Client Master

  • The credential count

  • The credential scope

To validate your plugin catalog’s suitability for deployment to a Client Master:

  1. Access the relevant Client Master’s configuration in Operations Center.

  2. Scroll down to the Plugin Catalog property and select its Specify a plugin catalog for this master check box.

  3. Select the plugin catalog from the Catalog dropdown.
    This name is the displayName specified when the plugin catalog was defined.

  4. Click the Check Validity button.
    If your plugin catalog’s definition is suitable for deployment to the Client Master, the message The catalog is compatible with the master is displayed.
    Plugin Catalog validity check

  5. ( Optional ) Scroll down the page and click the Save or Apply button to deploy the plugin catalog to the Client Master.

To validate your plugin catalog in this manner against all Client Masters that have been connected to your Operations Center instance, repeat this procedure on each Client Master.

Deploying a plugin catalog

A plugin catalog which has been added to Operations Center can be deployed to a configured Client Master (after optionally validating its suitability for deployment) using either the Operations Center interface or the Jenkins CLI tool.

Be aware that only one plugin catalog can be deployed to a Client Master at a time.

Deploying a plugin catalog using the Operations Center interface

  1. Access the relevant Client Master’s configuration in Operations Center.

  2. Scoll down to the Plugin Catalog property and select its Specify a plugin catalog for this master check box.

  3. Select the plugin catalog from the Catalog dropdown.
    This name is the displayName specified when the plugin catalog was defined.

  4. Scroll down the page and click the Save or Apply button to deploy the plugin catalog to the Client Master.

  5. ( Optional ) Verify the plugin catalog’s deployment to the Client Master.

Deploying a plugin catalog using the Jenkins CLI tool

This section assumes you have configured an alias for the Jenkins CLI tool.

  1. In a terminal/command prompt (window), enter the command:

    jenkins-cli plugin-catalog --push plugin-catalog-name --master client-master

    where plugin-catalog-name is the name of the plugin catalog specified when the plugin catalog was defined and client-master is the name of the Client Master configured on your Operations Center instance.
    For example, if you used the plugin catalog example above and have already added it to your Operations Center instance, then you would enter the command:

    jenkins-cli plugin-catalog --push additional-pipeline-plugins --master client-master

    If the command was successful, the Jenkins CLI tool returns the following JSON response:

    {
      "id": "additional-pipeline-plugins",
      "message": "Catalog applied correctly, it will be propagated in a few seconds",
      "status": "SUCCESS"
    }
    When using this Jenkins CLI command, the --push option must always be used in conjunction with the --master option. If you only use one of these options, the Jenkins CLI tool returns an error.
  2. ( Optional ) Verify the plugin catalog’s deployment to the Client Master.

Pushing a Plugin Catalog to all Client Masters using the Jenkins CLI tool

The following bash script will add a Plugin Catalog to Operations Center and push it to all the online Client Masters:

#!/usr/bin/env bash

JENKINS_CLI=jenkins-cli.jar
JENKINS_CJOC_URL=http://localhost:8080/
JENKINS_AUTH=admin:admin

if [ -z "$JENKINS_CJOC_URL" ]; then
    echo "Need to set environment variable JENKINS_CJOC_URL (Operations Center root URL)."
    exit 1
fi

if [ -z "$JENKINS_AUTH" ]; then
    echo "Need to set environment variable JENKINS_AUTH (format: 'userId:apiToken')."
    exit 1
fi


if [ -f "$JENKINS_CLI" ]
then
	echo "Using $JENKINS_CLI."
else
	wget -O "$JENKINS_CLI" $JENKINS_CJOC_URL/jnlpJars/jenkins-cli.jar
fi

java -jar $JENKINS_CLI -s $JENKINS_CJOC_URL -auth $JENKINS_AUTH  plugin-catalog --put < "java-web-catalog.json" | jq

java -jar $JENKINS_CLI -s $JENKINS_CJOC_URL -auth $JENKINS_AUTH list-masters | jq -r '.data.masters[] | select(.status == "ONLINE") | .fullName' | while read masterName; do
	java -jar $JENKINS_CLI -s $JENKINS_CJOC_URL -auth $JENKINS_AUTH plugin-catalog --push "java-web" --master "$masterName" | jq
done

Verifying a plugin catalog’s deployment to a Client Master

To validate that a plugin catalog has been successfully deployed to the Client Master:

  1. Ensure you are logged in to the Client Master as a user with the Administer permission.

  2. From the Client Master home page/Dashboard, click Manage Jenkins on the left.

  3. Click Beekeeper Upgrade Assistant.

  4. Click Plugin Catalog on the left (or the section’s More Info button on the right).
    Plugin Catalog on master
    If you see the message There is a plugin catalog installed, then the plugin catalog was deployed to the Client Master and the remaining details on this page should depict the plugin catalog that had been defined and added to Operations Center, and deployed to this Client Master.

    If you only just deployed a plugin catalog to the Client Master and you see No Plugin Catalog is installed at the moment, it may take a few minutes before the Client Master registers this deployment. Therefore, wait a few moments and refresh this page, or revisit this procedure at a later point in time.
  5. ( Optional ) For any Included plugins shown on this page, you can now install or update these plugins on the Client Master itself (if these plugins have not yet been installed/updated).

Removing a plugin catalog from a Client Master

A plugin catalog that has been deployed to a Client Master can be removed from it using either the Operations Center interface or the Jenkins CLI tool.

Removing a plugin catalog using the Operations Center interface

  1. Access the relevant Client Master’s configuration in Operations Center.

  2. Scoll down to the Plugin Catalog property and clear its Specify a plugin catalog for this master check box.

  3. Scroll down the page and click the Save or Apply button to confirm the removal of the plugin catalog from the Client Master.

  4. ( Optional ) Verify the plugin catalog’s removal from the Client Master.

Removing a plugin Catalog using the Jenkins CLI tool

This section assumes you have configured an alias for the Jenkins CLI tool.

  1. In a terminal/command prompt (window), enter the command:

    jenkins-cli plugin-catalog --remove client-master

    where client-master is the name of the Client Master configured on your Operations Center instance.
    For example, if you have already added a plugin catalog to your Operations Center instance and deployed it to a Client Master whose name is client-master, then you would enter the command:

    jenkins-cli plugin-catalog --remove client-master

    If the command was successful, the Jenkins CLI tool returns the following JSON response:

    {
      "id": "client-master",
      "message": "Catalog removed",
      "status": "SUCCESS"
    }
  2. ( Optional ) Verify the plugin catalog’s deployment to the Client Master.

Verifying a plugin catalog’s removal from a Client Master

To validate that a plugin catalog has been successfully removed from a Client Master:

  1. Ensure you are logged in to the Client Master as a user with the Administer permission.

  2. From the Client Master home page/Dashboard, click Manage Jenkins on the left.

  3. Click Beekeeper Upgrade Assistant.

  4. Click Plugin Catalog on the left (or the section’s More Info button on the right).
    If you see the message No Plugin Catalog is installed at the moment, then the plugin catalog was removed from the Client Master.

    If you only just removed a plugin catalog from the Client Master and you see There is a plugin catalog installed, it may take a few minutes before the Client Master registers this removal. Therefore, wait a few moments and refresh this page, or revisit this procedure at a later point in time.

Removing a plugin catalog from Operations Center

A plugin catalog added to your Operations Center instance can be removed from it using the Jenkins CLI tool. This section assumes you have configured an alias for the Jenkins CLI tool.

A plugin catalog can only be removed from your Operations Center instance when no Client Masters are using the plugin catalog.
  • In a terminal/command prompt (window), enter the command:

    jenkins-cli plugin-catalog --delete plugin-catalog-name

    where plugin-catalog-name is the name of the plugin catalog specified when the plugin catalog was defined.
    For example, if you used the plugin catalog example above, then you would enter the command:

    jenkins-cli plugin-catalog --delete additional-pipeline-plugins

    If the command was successful, the Jenkins CLI tool returns the following JSON response:

    {
      "id": "additional-pipeline-plugins",
      "message": "Catalog [additional-pipeline-plugins] successfully deleted",
      "status": "SUCCESS"
    }

Note:

  • If the Jenkins CLI tool returns an error like:

    {
      "message": "Catalog [additional-pipeline-plugins] is being used by the following master/s: [client-master]. It can not be deleted",
      "status": "FAILURE"
    }

    then the plugin catalog is still being used by at least one Client Master (in this example, a Client Master named client-master). If the plugin catalog is being used by multiple Client Masters, then each Client Master is listed in the error response. To rectify this issue, remove the plugin catalog from each Client Master listed in the error response and then try following this procedure again to remove the plugin catalog from your Operations Center instance.