Creating projects based on GitHub repository structure

6 minute read

The GitHub Branch Source plugin allows you to create a new project based on the repository structure from one or more GitHub users or organizations. You can either:

  • Use a Multibranch Pipeline to import a single repository’s branches, pull requests, and tags as Pipeline projects.

  • Use a GitHub Organization project to import all or a subset of repositories belonging to a GitHub user or organization as "Multibranch Pipeline" projects.

Multibranch Pipeline projects for a repository

The Multibranch Pipeline project scans a repository, creating the Pipeline projects for branches, pull requests, and/or tags, depending on the criteria you specify. To define a project you create a Pipeline script in a Jenkinsfile in the root directory of a branch in the project. It is not necessary to use exactly the same Pipeline script in each branch. Instead you can have different Pipeline scripts defined in different branches, or add and remove Jenkinsfiles depending on your use cases. After the project is created, Jenkins immediately runs that project. Removing a Pipeline script from a branch removes the associated projects from the Multibranch Pipeline. You can restrict the branches that are examined by using a regular expression on the GitHub Multibranch Pipeline configuration page. This plugin fits into the larger "Pipeline as Code" story; for more information, see Understanding and implementing Pipeline as Code.

CloudBees TV

Walkthrough for creating a GitHub Branch Source Multibranch Pipeline in CloudBees CI

How to Create a GitHub Branch Source Multibranch Pipeline in CloudBees CI

Working with GitHub pull requests

Multibranch pipeline projects have a wide range of other options for controlling which Pipeline projects are created and when they are run. For example, you can choose to build origin-repository pull requests in addition to, or instead of, building the plain branches; and you can choose to build pull requests without merging with the base branch, or in addition to building the merge product. The following features are included:

  • GitHub user or organization repository’s pull requests are treated as projects.

  • Pull request and branch build result status is reported back to GitHub.

When the proper webhooks are configured, Jenkins reports the status of the pull request’s projects to GitHub. The status is reported as follows:

  • Success - the job was successful

  • Failure - the job failed and the pull request is not ready to be merged

  • Error - something unexpected happened; example: the job was aborted in Jenkins

  • Pending - the job is waiting in the build queue

Example Multibranch Pipeline project

This example uses the following environment to create a Multibranch Pipeline.

  • Jenkins Controller 2.263.4

  • An agent connected to that controller (labeled "Linux").

  • Tunnel set up via ngrok to send webhooks from GitHub to the Jenkins Controller.

  • A repository for the application to get data for the Multibranch project. In this example we are using the Multibranch-sample-app repository.

To create a Multibranch Pipeline complete the following steps.

  1. Ensure that you have authenticated credentials for your repository. In this example we use the GitHub app for authentication. For step-by-step instructions on creating a GitHub application, see Using GitHub App authentication. For instructions on adding the credentials to your repository, see Using GitHub App authentication - Adding the Jenkins credential.

  2. Go to the Jenkins dashboard and click New Item.

  3. In the Enter an item name field, type in the name for the Multibranch Pipeline project that you want to create. For this example let us name the project sample-multibranch.

    enter an item name
  4. Scroll down and select Multibranch Pipeline.

  5. Click OK. The General settings page is displayed.

  6. Scroll down to Branch Sources. Click on the Add source dropdown and select GitHub.

  7. Select the credentials for your repository from the Credentials dropdown. These are the credentials you created for the GitHub app.

  8. Provide the url for the repository. In this example we are using the multibranch-sample-app repository. Your url should look something like this

  9. Click Save.

    Ensure that you have a Jenkinsfile in the root of your repository as the build is configured to look for a Jenkinsfile in the root of a branch.

You will notice that the webhook has been received by the Jenkins controller and is being processed.

If you now create a new branch in your repository, that branch will be automatically added to the project that you created. A PR associated with this project will also be created automatically.

Creating a GitHub Organization project

This plugin creates a new project type, a GitHub Organization. The GitHub organization project creates a Multibranch Pipeline for each repository in an organization. This Multibranch Pipeline then creates a separate project for each branch that contains a Jenkinsfile in the top level directory. To use this plugin, you must have GitHub credentials with the appropriate access level to your project (see the section on credentials below) installed in Jenkins. Set up your GitHub Organization as follows:

  1. From the Jenkins dashboard select New Item > GitHub Organization . Enter an item name and select OK .

    jenkins github org
  2. On the Projects tab, enter the details for GitHub Organization .

    github org projects tab
  3. Select or add new Credentials.

    These credentials must have sufficient permissions to at least see the repositories and their contents for the intended user or organization.
  4. In the Owner field, specify a username or an organization name.

  5. Save, and wait for the Scan Organization to run. The project’s progress is displayed on the left-hand side.

    scan org log
  6. You may need to refresh the page after it runs to see your repositories. Each of the repositories that you have imported is represented by a Multibranch Pipeline project. The configuration of these projects is controlled by the parent GitHub Organization project. The imported repository is represented as a folder containing a project for each branch with a Pipeline script defined in a Jenkinsfile in the root directory. You cannot modify the settings on these folders.

Selecting GitHub Organization project credentials

There are two different types of credentials for the GitHub organization:

  • scan credentials: used to access the GitHub API

  • checkout credentials: used to clone the repository from GitHub

You can use different credentials for each as long as they have the appropriate permissions. There is an anonymous option for checkout. Regardless, you should create a GitHub access token to use to avoid storing your password in Jenkins and prevent any issues when using the GitHub API. When using a GitHub access token, you must use standard Username with password credentials, where the username is the same as your GitHub username and the password is your access token.

Managing your GitHub Connections

GitHub rate limiting

GitHub limits the number of API calls that an account can make within a particular time frame, also known as the "rate limit". The GitHub Branch Source plugin tracks the rate limit and lets administrators specify a strategy for apportioning their API call budget.

To configure a different rate limit strategy (for example to have Jenkins attempt to evenly distribute GitHub API requests), go to Configure System in the Manage Jenkins settings.

From the Configure System screen, two settings related to GitHub API requests can be configured:

  • Normalize API requests - Attempt to evenly distribute GitHub API requests. This strategy can be helpful when several pipelines are managed by multiple users or if there are several concurrent builds, making it difficult to know when a pipeline has been paused because it exceeded the default GitHub API usage limit.

  • Throttle at/near rate limit - Restrict GitHub API requests only when near or above the rate limit. This strategy may be preferred over "Throttle for Normalize" if there are relatively few/infrequent queries to the GitHub API.

Using build triggers and webhooks

You can change the build configuration to perform the Scan Organization at different times. The default setting scans your GitHub repository for changes if it has not received any change notifications from GitHub within the past day. This scan will catch the repositories you’ve added or removed from the managed GitHub repository. You can always force the Scan Organization to run from the GitHub Organization page.

While the build triggers are often enough, you can set up webhooks to automatically trigger builds when changes are pushed to your GitHub repositories.

To do this you must have a GitHub login with a token.
To set up a webhook:
  1. Navigate to the main configuration settings page by selecting Manage Jenkins > Configure System.

  2. In the GitHub section, add a GitHub Server with your credentials.

  3. If you need a token, generate one by selecting the Advanced button next to the notebook and pencil, then selecting the Manage additional GitHub actions button next to Additional actions and select Convert login and password to token.

You can also configure this manually through GitHub by registering the URL provided in the help section of the server config.

In August 2020, the Jenkins project voted to replace the term master with controller. We have taken a pragmatic approach to cleaning these up, ensuring the least amount of downstream impact as possible. CloudBees is committed to ensuring a culture and environment of inclusiveness and acceptance - this includes ensuring the changes are not just cosmetic ones, but pervasive. As this change happens, please note that the term master has been replaced through the latest versions of the CloudBees documentation with controller (as in managed controller, client controller, team controller) except when still used in the UI or in code.