Creating projects based on GitHub repository structure

5 minute read

CloudBees no longer supports CloudBees Jenkins Distribution as of February 24, 2021. Please refer to the following step-by-step documentation for Migrating from CloudBees Jenkins Distribution to Jenkins LTS. The increased alignment between CloudBees Jenkins Distribution and Jenkins means users will experience zero impact to Pipeline execution. Existing customers can also contact CloudBees Support to help ensure a smooth transition.

Please see About the CloudBees Jenkins Distribution retirement for more information.

The GitHub Branch Source plugin allows you to create a new project based on the repository structure from one or more GitHub users or organizations. You can either:

  • Import all or a subset of repositories as jobs into the workspace from a GitHub user or organization

  • Import a single repository’s branches as jobs from a GitHub user or organization

The GitHub Organization project scans the repository, importing the Pipeline jobs it identifies according to your criteria. To define a job, you create a Pipeline script in a Jenkinsfile in the root directory of the project or branch. It is not necessary to use the exact same Pipeline script in each branch; instead you can have different Pipeline scripts defined in project branches, or add and remove Jenkinsfiles depending on your use cases. After a project is imported, Jenkins immediately runs a job. Removing a Pipeline script from a branch or from the entire project, removes the associated jobs from the view. You can restrict imports using a regular expression on the GitHub Organization configuration page. This plugin fits into the larger "Pipeline as Code" story; for more information, see Understanding and implementing Pipeline as Code.

The GitHub Branch Source plugin creates a folder for each repository. It then creates a separate job for each branch which contains a Jenkinsfile in the top level directory. The Jenkinsfile contains a Pipeline script outlining the steps of the job. Scripts can be different for each branch.

Importing GitHub pull requests

The GitHub Branch Source plugin allows the project to import externally originated pull requests containing a Pipeline script. The following features are included:

  • GitHub user or organization repository’s pull requests are treated as jobs

  • Pull request build result status reported back to GitHub

Pull requests will be added to Jenkins as long as the pull request originates from a remote repository, contains a Pipeline script in a Jenkinsfile, and is mergeable.

When the proper webhooks are configured, Jenkins will report the status of the pull request’s job to GitHub. The status is reported as follows:

  • Success - the job was successful

  • Failure - the job failed and the pull request is not ready to be merged

  • Error - something unexpected happened; example: the job was aborted in Jenkins

  • Pending - the job is waiting in the build queue

GiHub API requests

GitHub limits the number of API calls that an account can make within a particular timeframe, also known as the "rate limit". The GitHub Branch Source plugin tracks the rate limit and lets administrators specify a strategy for apportioning their API call budget.

To configure a different rate limit strategy (for example to have Jenkins attempt to evenly distribute GitHub API requests), go to Configure System in the Manage Jenkins settings.

From the Configure System screen, two settings related to GitHub API requests that can be configured:

  • Normalize API requests - Attempt to evenly distribute GitHub API requests. This strategy can be helpful when there are several pipelines managed by multiple users or if there are several concurrent builds, making it difficult to know when a pipeline has been paused due to exceeding the default GitHub API usage limit.

  • Throttle at/near rate limit - Restrict GitHub API requests only when near or above rate limit. This strategy may be preferred over "Throttle for Normalize" if there are relatively few/infrequent queries to the GitHub API.

Creating a GitHub Organization

This plugin creates a new project type, a GitHub Organization. To use this plugin, you must have GitHub credentials with the appropriate access level to your project (see the section on credentials below) installed in Jenkins. Set up your GitHub Organization as follows:

From the Jenkins dashboard select New Item > GitHub Organization. Enter an item name and select OK.

On the Projects tab, enter the details for GitHub Organization.

Select or add new Credentials.

These credentials should allow you to at least see the contents of your targeted repository.

In the Owner field, specify a username or an organization name.

Save, and wait for the Scan Organization to run. Job progress is displayed to the left hand side.

You may need to refresh the page after it runs to see your repositories.

A this point, you are finished with basic project configuration. You can now explore your imported repositories, configuring different settings on each of those folders if needed. You can also investigate the results of the jobs run as part of the Scan Organization. Each of your imported repositories is a folder containing a project for each branch with a Pipeline script defined in a Jenkinsfile in the root directory. You cannot modify the settings on these folders.

Selecting GitHub Organization credentials

There are two different types of credentials for the GitHub organization:

  • scan credentials: used to access the GitHub API

  • checkout credentials: used to clone the repository from GitHub

You can use different credentials for each as long as they have the appropriate permissions. There is an anonymous option for checkout. Regardless, you should create a GitHub access token to use to avoid storing your password in Jenkins and prevent any issues when using the GitHub API. When using a GitHub access token, you must use standard Username with password credentials, where the username is the same as your GitHub username and the password is your access token.

Using build triggers and webhooks in a GitHub Organization

You can change the build configuration to perform the Scan Organization at different times. The default setting will scan your GitHub repository for changes if it has not received any change notifications from GitHub within the past day. This scan will catch the repositories you’ve added or removed from the managed GitHub repository. You can always force the Scan Organization to run from the GitHub Organization page.

While the build triggers are often enough, you can set up webhooks to automatically trigger builds when changes are pushed to your GitHub repositories.

To do this you must have a GitHub login with a token.
To set up a webhook:
  1. Navigate to the main configuration settings page by selecting Manage Jenkins > Configure System.

  2. In the GitHub section, add a GitHub Server with your credentials.

  3. If you need a token, generate one by selecting the Advanced button next to the notebook and pencil, then selecting the Manage additional GitHub actions button next to Additional actions and select Convert login and password to token.

You can also configure this manually through GitHub by registering the URL provided in the help section of the server config.

Controlling what is built in a GitHub Organization

By default, Jenkins will build any branches it finds in the origin repository (the actual repository found in your organization). Also, any pull requests that have been filed from forks (clones in user accounts) will also be built; Jenkins will try to merge the pull request with the base (target) branch on every commit, to check for possible integration conflicts. Pull requests filed from the origin repository are skipped as these are already getting built as plain branches.

To customize this behavior, as of version 1.8 of the plugin, you can add Behaviors to the GitHub Organization configuration.

In particular, you can choose to build origin-repository pull requests in addition to, or instead of, building the plain branches; and you can choose to build pull requests without merging with the base branch, or in addition to building the merge product.