GitHub webhook triggering is delayed

Issue

Builds that should be triggered by GitHub webhooks are intermittently delayed for varying amounts of time. The jobs do not start or even enter the build queue until sometime after the GitHub change and webhook have been sent. Triggering the builds manually works immediately.

Environment

Resolution

In environments that have large numbers (in the thousands) of GitHub Branch Source jobs configured, it is possible for Jenkins controllers to become impacted by GitHub’s API request rate limit. This limit is configured in the GitHub service, not in Jenkins, and it defines a maximum number of API requests that any given GitHub user can make per hour. There are numerous background tasks that Jenkins performs which use the GitHub API, and when the API rate limit is reached these tasks back up in an internal buffer. This buffer is also used to process incoming GitHub webhooks for job triggering, so if there is a lot of work in the buffer waiting to be processed, it causes delays in processing webhooks.

To confirm that you are impacted by this, open a Multibranch job and click on the Multibranch Pipeline Events link in the left-side navigation. If you are hitting the GitHub rate limit, you will see log entries like this: Jenkins-Imposed API Limiter: Current quota for GitHub API usage has 5576 remaining (109 over budget). Next quota of 30000 in 11 min. Sleeping for 41 sec. You may have to look at the event log for a few different jobs before you see these entries, since not all jobs may be trying to use the API at any given time.

There are several possible ways to mitigate this problem. Depending on your circumstances, you may need to use more than one of them. If load on your GitHub servers will permit it, you could increase the rate limit itself. However this increase would apply to all users of the environment, so this may not be desirable. If you have multiple Jenkins controllers all integrating with one GitHub instance, and the controllers all share a single GitHub user for authentication, you can create separate GitHub users for each Jenkins controller so that each one has its own API request quota.

You may need to reduce the API request load coming from Jenkins. On controllers with large numbers of jobs, one of the best ways to do this is to increase the time interval between the branch indexing scans that every Multibranch job runs. These scans only serve one function, which is to synchronize the list of branches Jenkins knows about with the branches actually in GitHub. Normally the branch scan should find no changes, because when a branch is created or deleted a webhook from GitHub tells Jenkins to add or delete its job. The indexing scan serves only to catch any case when a branch add or remove webhook event was somehow lost. This is a very rare occurrence normally, so it is safe to increase this scan interval. You can always run a manual scan if it happens to be necessary. Note that users sometimes erroneously change this setting to a low/frequent value for their jobs, thinking that it will make them trigger more quickly; in fact this creates more load on the API and makes the triggering problem worse. By default the scan interval is set to once per day; it can be increased to every 7, 14, or 28 days. To set it globally for all jobs, a Groovy script can be run from the script console.

Note: the script will change all existing Multibranch jobs; new jobs will still be created with the default 1 day indexing interval unless changed by a user.

Groovy script for changing indexing interval globally:

//By default the script just prints out the current indexing intervals for all jobs.
//The function at the bottom can be called to set the interval to a new value.
//To call the function, change the two `each` blocks as follows:
// .each { setInterval(folder) }

import com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger
import jenkins.model.Jenkins
import jenkins.branch.OrganizationFolder

println "Organization Items\n-------"
Jenkins.instance.getAllItems(jenkins.branch.OrganizationFolder.class).each { folder -> folder.triggers
       .findAll { k,v -> v instanceof com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger }
       .each { k,v -> println "Folder name: ${folder.fullName}, Interval: ${v.getInterval()}" }
}

println "Multibranch Items\n-------"
Jenkins.instance.getAllItems(org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject.class).each { folder -> folder.triggers
       .findAll { k,v -> v instanceof com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger }
       .each { k,v -> println "Folder name: ${folder.fullName}, Interval: ${v.getInterval()}" }
}

return

//Acceptable values for triggers can be found here:
//https://github.com/jenkinsci/cloudbees-folder-plugin/blob/master/src/main/java/com/cloudbees/hudson/plugins/folder/computed/PeriodicFolderTrigger.java#L241
def setInterval(folder) {
  println "[INFO] : Updating ${folder.name}... "
  folder.getTriggers().find {triggerEntry ->
    def key = triggerEntry.key
    if (key instanceof PeriodicFolderTrigger.DescriptorImpl){
      println "[INFO] : Current interval : " + triggerEntry.value.getInterval()

      // Set the desired interval here
      def newInterval = new PeriodicFolderTrigger("28d")

      folder.addTrigger(newInterval)
      folder.save()
      println "[INFO] : New interval : " + newInterval.getInterval()
    }
  }
}