Managing Pipeline Template Catalogs in bulk

Managing Pipeline Template Catalogs across a large number of Jenkins controllers using the graphical user interface (GUI) is time consuming and prone to human error due to the repetitive nature of the task.

Using the provided CLI commands allows the administrator to automate the management of Pipeline Template Catalogs across multiple controllers, which reduces efforts and consistency across all development teams.

There are two provided CLI commands for working with Pipeline Template Catalogs, listing and creating/updating.

Listing all Pipeline Template Catalogs

Command

java -jar jenkins-cli.jar -s http://URL_TO_JENKINS pipeline-template-catalogs > output.json

You can include an -auth option or environment variables if you are using HTTP. You must use -user if you are using SSH. See Using the CLI Client for more information.

If you do not include > output.json - the JSON will print out in the console. Otherwise if you include it and you have a file with that name, the JSON will replace the contents of the file. If you do not already have a file with that name, the file will be created.

Success

You will receive a response with a list of imported Pipeline Template Catalogs in JSON format. Information about the Pipeline Template Catalogs in the response includes: the SCM information, branch(es) specified, and how often Jenkins checks for updates to templates in the Pipeline Template Catalog.

Example output.json with one catalog using a git scm

[
  {
    "scm":{
      "$class":"GitSCMSource",
      "remote":"https://github.com/owner/catalog-repo.git",
      "credentialsId":"git-repo-credentials"
    },
    "branchOrTag":"main",
    "updateInterval":"2d"
  }
]▼

Failure

You will receive an exception message dependent on what the issue involves and a status of "FAILURE".

{
  "message": "example failure message",
  "status": "FAILURE"
}▼

Creating or updating all Pipeline Template Catalogs

CloudBees recommends creating a backup copy of your Pipeline Template Catalogs before creating or updating. When you upload a JSON file using the put argument on the command, the JSON file of new catalogs will replace the current imported catalogs, which could result in accidental deletion of Pipeline Template Catalogs.

Command

java -jar jenkins-cli.jar pipeline-template-catalogs --put < input.json

You can include an -auth option or environment variables if you are using HTTP. You must use -user if you are using SSH. See Using the CLI Client for more information.

The input.json file in the command above is a sample file name - replace it with the name of the file you are uploading.

Success

You will receive a message stating the Pipeline Template Catalogs have imported successfully and the status will be "SUCCESS".

{
  "message": "Successfully imported Pipeline Template Catalogs.",
  "status": "SUCCESS"
}▼

Failure

You will receive an exception message dependent on what the issue involves and a status of "FAILURE".

{
  "message": "example failure message",
  "status": "FAILURE"
}▼

Common errors

Often failures occur with the put command because of invalid or missing data. Some common examples include:

NullPointerException: Missing branchOrTag configuration - see below for examples of valid branchOrTag configurations
NullPointerException: Missing scm configuration - see below for examples of valid scm configurations
Restoring a deleted template to a job: When you have a job based on a template, if the template is deleted, the job goes into a "detached" state, where it can’t be built anymore, because the template doesn’t exist. For Pipeline Template Catalogs, if you recreate a template with the same name, for example by reconfiguring the deleted catalog, then the template will be reattached to the job and the job will be able to be built again.

Catalog properties

The following are catalog properties used to configure Pipeline Template Catalogs.

`scm`

The scm property is required. There are three scm options available: GitSCMSource, GitHubSCMSource, and BitbucketSCMSource. Below are the properties required for the corresponding scm values. NOTE: In order for these configurations to work users must have the corresponding SCM plugin ("Git", "GitHub Branch Source", or "Bitbucket Branch Source") installed for their Jenkins instance.

It is recommended to configure Bitbucket using GitSCMSource rather than BitbucketSCMSource because it is simpler to configure.

`GitSCMSource`

The $class property is required. Use GitSCMSource for generic Git and Bitbucket repositories.
The remote property is required and is the git repo URL.
The credentialsId property is optional, but required to access private repos. This property is configured in the CloudBees CI UI, so even if you are using the CLI commands, you have to go into the CloudBees CI UI and create a credential with that name to be able to reference it from the CLI. See Configuring Credentials for more information.

`GitHubSCMSource`

The $class property is required. Use GitHubSCMSource for GitHub repositories.
The apiUrl property is optional. Use this property for GitHub Enterprise accounts.
The credentialsId property is optional, but recommended to increase API limits and required to access private repos and/or configure webhooks. This property is configured in the CloudBees CI UI, so even if you are using the CLI commands, you have to go into CloudBees CI UI and create a credential with that name to be able to reference it from the CLI. See Configuring Credentials for more information.
The repoOwner property is required. This property is the user or organization name attached to the repository.
The repository property is required. This property is the name of the repository.

`BitbucketSCMSource`

It is recommended that you use GitSCMSource instead of BitbucketSCMSource; however, if you want to use webhooks for immediate notification of changes to templates, which will take precedence over updateInterval, you must use BitbucketSCMSource as your scm.

The $class property is required. Use BitBucketSCMSource for Bitbucket repositories, if you want to use webhooks for immediate notification of changes to templates, which will take precedence over updateInterval.
The credentialsId property is optional, but recommended to increase API limits and required to access private repos and/or configure webhooks. This property is configured in the CloudBees CI UI, so even if you are using the CLI commands, you have to go into CloudBees CI UI and create a credential with that name to be able to reference it from the CLI. See Configuring Credentials for more information.
The serverUrl property is required and is the URL of the Bitbucket server.
The repoOwner property is required. This property is the user or organization name attached to the repository.
The repository property is required. This property is the name of the repository.
The traits property is optional. If you want to include specific tags or branches though you need to include traits.
- The $class property is required and can either be com.cloudbees.jenkins.plugins.bitbucket.BranchDiscoveryTrait or com.cloudbees.jenkins.plugins.bitbucket.TagDiscoveryTrait.
- The strategyId property is required if you are using the com.cloudbees.jenkins.plugins.bitbucket.BranchDiscoveryTrait property as it specifies whether you want to include:
  - All branches (no PRs): 1
  - Only PRs: 2
  - All branches (including PRs): 3

`branchOrTag`

The branchOrTag property is required. Valid values include branch names, tag names, and commit hashes. For example, "default", "2.61.x", "product-abc", or "d8937070209181a51d".

`updateInterval`

The updateInterval property is also optional and defaults to 1 day.
The updateInterval value determines how often CloudBees CI checks the source code repository to see if there are any changes to your Pipeline template catalog.
In most cases, you will not need to change this value since many source code management systems use webhooks to automatically send notifications when a source code repository is updated.
CloudBees CI will not poll the source code repository for updates in this time interval if events have already been pushed to the catalog on this CloudBees CI controller.
Valid options are '15m', '20m', '25m', '30m', '1h', '2h', '4h', '8h', '12h', '1d', '2d', '7d', '14d', and '28d'. If you specify an invalid value, 15 minutes will be used instead.
If you are using GitSCMSource, then you might want to set the property to a lower value so that you receive updates more quickly.
A user with MANAGE_CATALOGS permission can trigger a new scan manually if they want to bypass the setting.

`parentName`

This is the folder (for team controllers only) where the catalogs will be available. The value it contains can be changed if you have got another team controllers folders.

If you have the field set to run too frequently, it could cause performance issues if you have a large number of catalogs or a really large catalog.

Example JSON file format

The following example shows how to configure a Pipeline Template Catalogs JSON file for import using Git, GitHub, and Bitbucket.

[
   {
     "scm":{ (1)
       "$class":"GitSCMSource",
       "remote":"https://github.com/car-roll/my-pipeline-catalog.git",
       "credentialsId":"my-git-credentials"
     },
     "branchOrTag":"main",
     "updateInterval":"2d",
     "parentName": "/team-five"
   },
   {
     "scm":{ (2)
       "$class":"GitHubSCMSource",
       "credentialsId":"my-github-credentials",
       "repoOwner":"car-roll",
       "repository":"my-pipeline-catalog"
     },
     "branchOrTag":"main",
     "updateInterval":"8h"
   },
   {
     "scm":{ (3)
       "$class":"GitSCMSource",
       "remote":"https://bitbucket.org/carchtest_team1/team-repo1.git",
       "credentialsId":"my-bitbucket-credentials"
     },
     "branchOrTag":"main",
     "updateInterval":"1d"
   },
   {
     "scm": { (4)
       "$class": "BitbucketSCMSource",
       "credentialsId": "my-bitbucket-credentials",
       "serverUrl": "https://bitbucket.org",
       "repoOwner": "myCompanyOrTeam",
       "repository": "repo-name",
       "traits": [
         {
           "$class": "com.cloudbees.jenkins.plugins.bitbucket.BranchDiscoveryTrait",
           "strategyId": 1
         },
         {
           "$class": "com.cloudbees.jenkins.plugins.bitbucket.TagDiscoveryTrait"
         }
       ]
    },
    "branchOrTag": "main",
    "updateInterval": "2d"
    }
]▼

1	Git example
2	GitHub example
3	Bitbucket example - without webhooks
4	Bitbucket example - with webhooks