Installing non-CAP plugins with plugin catalogs

A plugin catalog is a configuration file that specifies plugins and their associated versions.

The Beekeeper Upgrade Assistant uses a plugin catalog to widen the acceptable scope of plugins beyond those defined by the CloudBees Assurance Program. Your plugin catalog can also specify a Maven 2 artifact repository from which to download plugins.

Once a plugin catalog is defined and is added to your operations center instance, it can then be deployed to your connected client controller or managed controller instances.

When to use a plugin catalog

A plugin catalog is useful when you want to:

Ensure that certain plugins are kept at a consistent version.
Provide an alternative location for plugins if Jenkins is not connected to the internet.
Install and maintain plugins that are not in CloudBees Assurance Program, for example custom plugins or older versions of plugins.

What happens when a plugin catalog is configured

The following changes occur when you configure a plugin catalog:

The Jenkins Update Center contains the plugins that are defined in the plugin catalog. Any versions defined in the plugin catalog override the versions in the Update Center.
If you install a plugin version that is defined in a plugin catalog, that plugin version is managed by the plugin catalog.

Defining a plugin catalog

Before you can add a plugin catalog to your operations center instance, you must first define it.

A plugin catalog is defined in JSON file format, which in turn defines one or more plugin configurations. A plugin configuration defines the set of rules for one or more plugins (some potentially dependencies) and their versions that can be installed to your operations center instance, along with the CloudBees Assurance Program-defined plugins.

To create this file, using your favorite text editor, copy and paste the unannotated empty template below into a new file and save it with an appropriate file name, similar to that of the name or displayName value.

The following JSON content is an example of the contents of a plugin catalog file, whose annotations are explained underneath. An appropriate name for this plugin catalog file might be
additional-pipeline-plugins-catalog.json. Use this information to help complete your own plugin catalog file.

Annotated plugin catalog file

{
    "type" : "plugin-catalog",(1)
    "version" : "1",(2)
    "name" : "additional-pipeline-plugins",(3)
    "displayName" : "Additional Pipeline plugins",(4)
    "configurations" : [(5)
        {
            "description" : "Additional Pipeline plugins for controllers versions 2.61 and 2.73", (6)
            "prerequisites": {(7)
                "productVersion": "[2.60, 2.73]"
            },
            "includePlugins" : {(8)
                "tier3-plugin": {
                    "groupId": "com.my-company.jenkins.plugins",(9)
                    "version" : "1.8.2"
                },
                "my-corporate-plugin": {
                    "url" : "https://download.my-company.com/jenkins/my-corporate-plugin-3.2.1.hpi"
                },
                "my-other-corporate-plugin": {
                    "sha1" : "SC/TzSN+eOrewaDZJZyzLpIQV7E=",(10)
                    "url" : "https://download.my-company.com/jenkins/my-other-corporate-plugin-1.2.3.hpi"(11)
                }
            }
        },
        {(12)
            "description" : "Additional Pipeline plugins for controllers versions 2.89+",
            "prerequisites": {
                "productVersion": "[2.89)"
            },
            "includePlugins" : {
                "tier3-plugin": {
                    "groupId": "com.my-company.jenkins.plugins",
                    "version" : "1.8.3"
                },
                "my-corporate-plugin": {
                    "url" : "https://download.my-company.com/jenkins/my-corporate-plugin-3.4.0.hpi"
                },
                "my-other-corporate-plugin": {
                    "sha1" : "SC/TzSN+eOrewaDZJZyzLpIQV7E=",
                    "url" : "https://download.my-company.com/jenkins/my-other-corporate-plugin-1.3.0.hpi"
                }
            }
        }
    ],
    "settings": {(13)
        "httpCredentialsProvider": {(14)
            "credentials": [(15)
                {
                    "authenticationScope": "ANY",(16)
                    "id": "download-server-credentials"(17)
                }
            ]
        },
        "repository": { (18)
            "layout": "maven2",(19)
            "url": "https:#repo.my-company.com/content/repositories/dev-connect"
        }
    }
}▼

1	`type` (required) - must have the fixed value `plugin-catalog`, which defines this JSON file as a plugin catalog file.
2	`version` (required) - must currently have the value `1`, which defines the current metadata format for the plugin catalog.
3	`name` (required) - the internal name used to identify the plugin catalog. Avoid using spaces in this name.
4	`displayName` (optional) - a human-readable name for the plugin catalog. This is the name of the plugin catalog that appears in the operations center interface.
5	`configurations` (required) - the plugin catalog can define one or more plugin configurations within this array, where each plugin configuration is represented by an individual element of this array. At least one plugin configuration is required.
6	`description` - provides more context regarding the configuration itself.
7	`prerequisites` (optional) - an object which allows the definition of the `productVersion` member, which defines a range of supported client controller or managed controller versions (using the Maven Version Range Specification) for the plugin configuration. If the `prerequisites` object (and hence its `productVersion` member) is omitted, then any client controller or managed controller version is supported.
8	`includePlugins` (optional) - an object which allows you to specify the ID values of one or more plugins to include as part of this plugin configuration. For each plugin (specified by its `plugin-id`), specify the pair made up of the supported groupId and version for the plugin or the URL from where download it.
9	`groupId` (optional) - provides the `groupId` for the plugin when its download will take place from a maven repository. If the `groupId` is not provided, this field will consider `org.jenkins-ci.plugins` as default value.
10	`sha1` (optional) - the SHA1 hash of the plugin that will be used to verify the downloaded plugin.
11	`url` (optional) - the download URL for the plugin.
12	(Optional) - an additional plugin configuration that provides support for different plugin versions (or plugins themselves), for different client controller or managed controller versions (specified in the `prerequisites` > `productVersion` member).
13	`settings` (optional) - the plugin catalog defines all parameters used in the download of the plugins.
14	`httpCredentialsProvider` (optional) - the plugin catalog defines a set of credentials using the interface CredentialsProvider.
15	`credentials` (required) - the array containing the credentials. Currently must contain only one element.
16	`authenticationScope` (required) - if the credentials are defined, must currently have the value `ANY`.
17	`id` (required) - the ID of a credential that must be defined in the client controller or managed controller where the plugin catalog will be installed. This credential will be used for plugin downloads from servers which require authentication.
18	`repository` (optional) - an object which allows the definition of a Maven 2 repository from which the plugins (defined in the `configurations` array above) can be downloaded. If a plugin’s `url` has not been specified within its relevant `configurations` section, then the client controller or managed controller will attempt to download the plugin from this `repository`‍'s `url`. Defining a repository is useful when you need to restrict a client controller’s or managed controller’s access to the Internet for security reasons (part of a process known as air gapping) but want the client controller or managed controller to install and update plugins from the repository, which can be achieved by setting up a proxy artifact repository.
19	`layout` (optional) - must currently have the value `maven2`. However, if this value is not defined, it is assumed to be `maven2` by default.

Copyable plugin catalog file

{
    "type" : "plugin-catalog",
    "version" : "1",
    "name" : "",
    "displayName" : "",
    "configurations" : [
        {
            "description" : "",
            "prerequisites": {
                "productVersion": ""
            },
            "includePlugins" : {
                "replace-with-appropriate-plugin-id": {
                    "groupId": "",
                    "version" : ""
                },
                "replace-with-other-appropriate-plugin-id": {
                    "sha1" : "",
                    "url" : ""
                }
            }
        }
    ],
    "settings": {
        "httpCredentialsProvider": {
            "credentials": [
                {
                    "authenticationScope": "ANY",
                    "id": ""
                }
            ]
        },
        "repository": {
            "layout": "maven2",
            "url": ""
        }
    }
}▼

Adding a plugin catalog

Once you have defined your plugin catalog, you can add it to your operations center instance using the Jenkins CLI tool. This section assumes you have configured an alias for the Jenkins CLI tool.

See the CLI user guide for more information on downloading and configuring the Jenkins CLI tool.

In a terminal/command prompt (window), change to the directory containing the plugin catalog file you defined in Defining a plugin catalog.
Enter the command:
```
jenkins-cli plugin-catalog --put < plugin-catalog-file.json▼
```
where plugin-catalog-file.json is the name of your plugin catalog file.
For example, if you used the plugin catalog example above and saved it to a file named
additional-pipeline-plugins-catalog.json, then you would enter the command:
```
jenkins-cli plugin-catalog --put < additional-pipeline-plugins-catalog.json▼
```
If the command was successful, the Jenkins CLI tool returns the following JSON response:
```
{
  "id": "additional-pipeline-plugins",
  "message": "Catalog updated correctly",
  "status": "SUCCESS"
}▼
```
You can now validate the plugin catalog’s suitability for deployment on a configured client controller or managed controller.
```
{
  "message": "Unexpected character ('}' ... [Source: ... line: 16, column: 14]",
  "status": "FAILURE"
}▼
```

Updating a plugin catalog on operations center

If one or more plugins (defined in a plugin catalog) have updates, you may need to specify these plugin updates in the plugin catalog and then update this plugin catalog on operations center.

This process is identical to adding the plugin catalog to operations center:

Modify the plugin catalog file defined previously (for example, with the updated plugin versions).
Re-add the plugin catalog to operations center.

You should re-validate and then re-deploy the updated plugin catalog to the client controllers or managed controllers on which the older version of this plugin catalog was deployed.

If you are unsure which client controllers or managed controllers the older version of the plugin catalog had been deployed to, verify the plugin catalog’s deployment on the relevant client controllers or managed controllers.

Validating a plugin catalog’s suitability for a client controller or managed controller

Once you have added a plugin catalog to operations center, you can verify this on the operations center interface when validating the plugin catalog’s suitability for deployment to a configured client controller or managed controller.

This validation process verifies:

That the client controller or managed controller version and plugin configuration satisfies exactly one plugin catalog definition
That plugin dependencies are resolved
That credentials referenced in the plugin catalog are defined in the client controller or managed controller
The credential count
The credential scope

To validate your plugin catalog’s suitability for deployment to a client controller or managed controller:

Access the relevant client controller’s or managed controller’s configuration in operations center.
Scroll down to the Plugin Catalog property and select its Specify a plugin catalog for this controller check box.
Select the plugin catalog from the Catalog dropdown.
This name is the displayName specified when the plugin catalog was defined.
Click the Check Validity button.
If your plugin catalog’s definition is suitable for deployment to the client controller or managed controller, the message The catalog is compatible with the controller is displayed.
( Optional ) Scroll down the page and click the Save or Apply button to deploy the plugin catalog to the client controller or managed controller.

To validate your plugin catalog in this manner against all client controllers or managed controllers that have been connected to your operations center instance, repeat this procedure on each client controller or managed controller.

Deploying a plugin catalog

A plugin catalog which has been added to operations center can be deployed to a configured client controller or managed controller (after optionally validating its suitability for deployment) using either the operations center interface or the Jenkins CLI tool.

Be aware that only one plugin catalog can be deployed to a client controller or managed controller at a time.

Deploying a plugin catalog using the operations center interface

Access the relevant client controller’s or managed controller’s configuration in operations center.
Scoll down to the Plugin Catalog property and select its Specify a plugin catalog for this controller check box.
Select the plugin catalog from the Catalog dropdown.
This name is the displayName specified when the plugin catalog was defined.
Scroll down the page and click the Save or Apply button to deploy the plugin catalog to the client controller or managed controller.
( Optional ) Verify the plugin catalog’s deployment to the client controller or managed controller.

Deploying a plugin catalog using the Jenkins CLI tool

This section assumes you have configured an alias for the Jenkins CLI tool.

In a terminal/command prompt (window), enter the command:

jenkins-cli plugin-catalog --push plugin-catalog-name --master client-controller▼

where plugin-catalog-name is the name of the plugin catalog specified when the plugin catalog was defined and client-controller is the name of the client controller or managed controller configured on your operations center instance.

For example, if you used the plugin catalog example above and have already added it to your operations center instance, then you would enter the command:

jenkins-cli plugin-catalog --push additional-pipeline-plugins --master client-controller▼

If the command was successful, the Jenkins CLI tool returns the following JSON response:

{
  "id": "additional-pipeline-plugins",
  "message": "Catalog applied correctly, it will be propagated in a few seconds",
  "status": "SUCCESS"
}▼

When using this Jenkins CLI command, the --push option must always be used in conjunction with the --master option. If you only use one of these options, the Jenkins CLI tool returns an error.

( Optional ) Verify the plugin catalog’s deployment to the client controller or managed controller.

Pushing a plugin catalog to all client controllers or managed controllers using the Jenkins CLI tool

The following bash script will add a Plugin Catalog to operations center and push it to all the online client controllers or managed controllers:

#!/usr/bin/env bash

JENKINS_CLI=jenkins-cli.jar
JENKINS_CJOC_URL=http://localhost:8080/
JENKINS_AUTH=admin:admin

if [ -z "$JENKINS_CJOC_URL" ]; then
    echo "Need to set environment variable JENKINS_CJOC_URL ({OC} root URL)."
    exit 1
fi

if [ -z "$JENKINS_AUTH" ]; then
    echo "Need to set environment variable JENKINS_AUTH (format: 'userId:apiToken')."
    exit 1
fi


if [ -f "$JENKINS_CLI" ]
then
	echo "Using $JENKINS_CLI."
else
	wget -O "$JENKINS_CLI" $JENKINS_CJOC_URL/jnlpJars/jenkins-cli.jar
fi

java -jar $JENKINS_CLI -s $JENKINS_CJOC_URL -auth $JENKINS_AUTH  plugin-catalog --put < "java-web-catalog.json" | jq

java -jar $JENKINS_CLI -s $JENKINS_CJOC_URL -auth $JENKINS_AUTH list-masters | jq -r '.data.masters[] | select(.status == "ONLINE") | .fullName' | while read masterName; do
	java -jar $JENKINS_CLI -s $JENKINS_CJOC_URL -auth $JENKINS_AUTH plugin-catalog --push "java-web" --master "$masterName" | jq
done▼

Verifying a plugin catalog’s deployment to a client controller or managed controller

To validate that a plugin catalog has been successfully deployed to the client controller or managed controller:

Ensure you are logged in to the client controller or managed controller as a user with the Administer permission.
From the client controller or managed controller home page/Dashboard, click Manage Jenkins on the left.
Click Beekeeper Upgrade Assistant.

Click Plugin Catalog on the left (or the section’s More Info button on the right).

Plugin Catalog on controller

If the message There is a plugin catalog installed is displayed, then the plugin catalog was deployed to the client controller or managed controller and the remaining details on this page should depict the plugin catalog that was defined and added to the operations center, and then deployed to this client controller or managed controller.

If you only just deployed a plugin catalog to the client controller or managed controller and you see No Plugin Catalog is installed at the moment, it may take a few minutes before the client controller or managed controller registers this deployment. Therefore, wait a few moments and refresh this page, or revisit this procedure at a later point in time.

( Optional ) For any Included plugins shown on this page, you can now install or update these plugins on the client controller or managed controller itself (if these plugins have not yet been installed/updated).

Removing a plugin catalog from a client controller or managed controller

A plugin catalog that has been deployed to a client controller or managed controller can be removed from it using either the operations center interface or the Jenkins CLI tool.

Removing a plugin catalog using the operations center interface

Access the relevant client controller’s or managed controller’s configuration in operations center.
Scoll down to the Plugin Catalog property and clear its Specify a plugin catalog for this controller check box.
Scroll down the page and click the Save or Apply button to confirm the removal of the plugin catalog from the client controller or managed controller.
( Optional ) Verify the plugin catalog’s removal from the client controller or managed controller.

Removing a plugin Catalog using the Jenkins CLI tool

This section assumes you have configured an alias for the Jenkins CLI tool.

In a terminal/command prompt (window), enter the command:
```
jenkins-cli plugin-catalog --remove client-controller▼
```
In this example, client-controller is the name of the client controller or managed controller configured on your operations center instance.

For example, if you have already added a plugin catalog to your operations center instance and deployed it to a client controller whose name is client-controller, then you would enter the command:
```
jenkins-cli plugin-catalog --remove client-controller▼
```
If the command was successful, the Jenkins CLI tool returns the following JSON response:
```
{
  "id": "client-controller",
  "message": "Catalog removed",
  "status": "SUCCESS"
}▼
```
( Optional ) Verify the plugin catalog’s deployment to the client controller or managed controller.

Verifying a plugin catalog’s removal from a client controller or managed controller

To validate that a plugin catalog has been successfully removed from a client controller or managed controller:

Ensure you are logged in to the client controller or managed controller as a user with the Administer permission.
From the client controller or managed controller home page/Dashboard, click Manage Jenkins on the left.
Click Beekeeper Upgrade Assistant.

Click Plugin Catalog on the left (or the section’s More Info button on the right).
If you see the message No Plugin Catalog is installed at the moment, then the plugin catalog was removed from the client controller or managed controller.

If you only just removed a plugin catalog from the client controller or managed controller and you see There is a plugin catalog installed, it may take a few minutes before the client controller or managed controller registers this removal. Therefore, wait a few moments and refresh this page, or revisit this procedure at a later point in time.

Removing a plugin catalog from operations center

A plugin catalog added to your operations center instance can be removed from it using the Jenkins CLI tool. This section assumes you have configured an alias for the Jenkins CLI tool.

A plugin catalog can only be removed from your operations center instance when no client controllers or managed controllers are using the plugin catalog.

In a terminal/command prompt (window), enter the command:

jenkins-cli plugin-catalog --delete plugin-catalog-name▼

In this example, plugin-catalog-name is the name of the plugin catalog specified when the plugin catalog was defined. For example, if you used the plugin catalog example above, then enter the following command:

jenkins-cli plugin-catalog --delete additional-pipeline-plugins▼

If the command is successful, the Jenkins CLI tool returns the following JSON response:

{
  "id": "additional-pipeline-plugins",
  "message": "Catalog [additional-pipeline-plugins] successfully deleted",
  "status": "SUCCESS"
}▼

If the Jenkins CLI tool returns an error like:

{
  "message": "Catalog [additional-pipeline-plugins] is being used by the following controller/s: [client-controller]. It can not be deleted",
  "status": "FAILURE"
}▼

then the plugin catalog is still being used by at least one client controller or managed controller (in this example, a client controller named client-master). If the plugin catalog is being used by multiple client controllers, then each client controller or managed controller is listed in the error response. To rectify this issue, remove the plugin catalog from each client controller or managed controller listed in the error response and then try following this procedure again to remove the plugin catalog from your operations center instance.

In August 2020, the Jenkins project voted to replace the term master with controller. We have taken a pragmatic approach to cleaning these up, ensuring the least amount of downstream impact as possible. CloudBees is committed to ensuring a culture and environment of inclusiveness and acceptance - this includes ensuring the changes are not just cosmetic ones, but pervasive. As this change happens, please note that the term master has been replaced through the latest versions of the CloudBees documentation with controller (as in managed controller, client controller, team controller) except when still used in the UI or in code.