GitHub: Webhook Troubleshooting

Symptoms

Jobs are not automatically triggered when after a Push or Pull request event in GitHub (and/or GitHub Enterprise)

Diagnostic/Treatment

Pre-condition : Jenkins (resp. GitHub) can reach out to GitHub (resp. Jenkins)

This troubleshooting requires from analysis of GitHub which generates a payload and Jenkins which parses it.

A. On GitHub

A.1 Webhook has not been generated or it is disabled (red cross)

Check that "GitHub-API-User" has Admin permissions for the repo. Or Owner role for Webhook at Organization level.
If GitHub Username and Password is used for GitHub plugin configuration, validate that the user has not enabled 2FA.
Important Notice GitHub is eliminating password-based API access (as mentioned in GitHub documentation), we strongly recommend you to switch your passwords so that you use the GitHub Access tokens or you use a different method to authenticate such as GitHub App authentication.

A.2 When the Webhook is generated

As explained in GitHub Testing Webhooks.

Click on Redeliver and evaluate the response and see if the Jenkins receive the payload in B.3 GitHub Hooks Problems Console and B.4 Dedicated Logger.

B. On Jenkins

B.1 Regenerate all Webhooks for your Jenkins instance

Go to Manage Jenkins > Configure System > GitHub plugin > Advance > Re-registers hooks for all jobs.

B.2 Validate Plugin Configuration

Identify which GitHub Server ("GitHub Server X") contains the repository/organization where webhook is failing.
For that server ("GitHub Server X"), validate its "GitHub API User" and its Rate limits by Manage Jenkins > GitHub plugin > "GitHub Server X" > Test Connection
```
 Verify credentials
 Credentials verified for user "GitHub-API-User", rate limit: 4994
```

3.Validate the configuration of $JENKINS_HOME/github-plugin-configuration.xml:

```xml
<github-plugin-configuration plugin="github@1.29.4">
  <configs>
    <github-server-config>
      <name>GITHUB</name>
      <apiUrl>https://api.github.com</apiUrl>
      <manageHooks>true</manageHooks>
      <credentialsId>GH-mock-ST</credentialsId>
      <clientCacheSize>20</clientCacheSize>
    </github-server-config>
  </configs>
  <hookSecretConfig>
    <credentialsId></credentialsId>
  </hookSecretConfig>
```

A parent <github-server-config> is need to each of the GitHub and GitHub Enterprise Servers if you would like the plugin to manage hooks for you.
For each server, you need to have configured a credential Secret Text GitHub API Token: <credentialsId>GH-example-APIToken-ST</credentialsId> or GitHub App private key
For each server, you need to have configured <manageHooks>true</manageHooks> to enable Re-registers hooks for all jobs

B.3 Validate Job Configuration

Review the plugin a jobs configurations as explained in GitHub WebHook Configuration for each of type of jobs.

B.4 GitHub Hooks Problems Console

Available from Manage Jenkins > GitHub Hooks Problems Console only and if only any of Automatic Webhooks has not been setup in GitHub.

This page shows problems with webhooks, and ignored projects.

Notes: Another warning messages also can be found at:

in Manage Jenkins
in the trigger configuration for For Non-Multibranch jobs

B.5 Check if the computed folder is at maximum indexing capacity

If you collect a support bundle with the Build queue option checked, then look at the buildqueue.md file in that support bundle, check if there is the string:

Why in queue: At maximum indexing capacity

This message is emitted by this code, when the indexing limit is reached.

To explain why this is relevant, it is because an Organization folder is a ComputedFolder, and ComputedFolders have an indexing limit, and if you have too many Organization folder items on a controller being processed concurrently (for example if you have a monolith controller), you can reach this limit, and job triggering for organization folders will be delayed.

The limit can be adjusted using this system property:

-Dcom.cloudbees.hudson.plugins.folder.computed.ThrottleComputationQueueTaskDispatcher.LIMIT=10

B.6 Dedicated Logger

Create a logger for GitHub in https://<JENKINS_URL>/log/ at the ALL level for (depending on the plugins you are using):

org.jenkinsci.plugins.github_branch_source For the GitHub Branch Source plugin (Multbranch and Organization folders from GitHub)
com.cloudbees.jenkins.plugins.bitbucket for the Bitbucket Branch Source plugin (Multbranch and Team/Project folders from Bitbucket)
org.jenkinsci.plugins.github and com.cloudbees.jenkins.GitHubWebHook for the GitHub plugin (webhooks sent to $JENKINS_URL/github-webhook/)
hudson.plugins.git.GitStatus for the Git plugin GitStatus class

Test Credentials

A similar output to this is expected:

Jul 01, 2016 12:51:38 PM FINE org.jenkinsci.plugins.github.webhook.GHEventHeader$PayloadHandler parse
Header X-GitHub-Event -> null
Jul 01, 2016 12:51:41 PM FINE org.jenkinsci.plugins.github.internal.GitHubLoginFunction applyNullSafe
Create new GH client with creds id 021643c7-ef89-4014-b835-6fab9add5165

Push payload

A similar output to this is expected:

Jul 01, 2016 12:32:03 PM FINE org.jenkinsci.plugins.github.webhook.GHEventHeader$PayloadHandler parse
Header X-GitHub-Event -> push
Jul 01, 2016 12:32:03 PM FINEST org.jenkinsci.plugins.github.webhook.GHEventPayload$PayloadHandler parse
Payload {"ref":"refs/heads/master","before":"a1a91ab98386e4eced8ce0720d5c44158e85a3db","after":"58831ba23954684d98c91fdd303b2c8cebc7c504","created":false,"deleted":false,"forced":false,"base_ref":null,"compare":"https://github.com/example-org/example-repo/compare/a1a91ab98386...58831ba23954","commits":

...

Jul 01, 2016 12:32:03 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received POST for
https://github.com/example-org/example-repo
Jul 01, 2016 12:32:03 PM FINE org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber$1 run
Considering to poke ZD-37633-CBGitHubPRBuilder
Jul 01, 2016 12:32:03 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber$1 run
Poked ZD-37633-CBGitHubPRBuilder

Pull Request payload

A similar output to this is expected:

Jul 01, 2016 2:33:23 PM FINE org.jenkinsci.plugins.github.webhook.GHEventHeader$PayloadHandler parse
Header X-GitHub-Event -> pull_request
Jul 01, 2016 2:33:23 PM FINEST org.jenkinsci.plugins.github.webhook.GHEventPayload$PayloadHandler parse
Payload {"action":"closed","number":2,"pull_request":{"url":"https://api.github.com/repos/example-org/example-repo/pulls/2","id":75972749,"html_url":"https://github.com/example-org/example-repo/pull/2","diff_url":"https://github.com/example-org/example-repo/pull/2.diff","patch_url":"https://github.com/example-org/example-repo-
...

B.5 GitHub Hook Log

Within the Job Page > GitHub Hook Log

B.6 Jenkins logs

The following traces illustrate successful build triggers on Jenkins logs:

Webhook register

Nov 09, 2017 6:18:43 PM org.jenkinsci.plugins.github.config.GitHubPluginConfig doReRegister
INFO: Called registerHooks() for 1 jobs
Nov 09, 2017 6:18:43 PM org.jenkinsci.plugins.github.webhook.WebhookManager$1 run
INFO: GitHub webhooks activated for job myJobExample with [GitHubRepositoryName[host=github.com,username=example-org,repository=example-repo]] (events: [PUSH, REPOSITORY])

In case the webhook was sucessfully, Jenkins receives a PING event from repo webhook so a similar trace like the following would appear:

Nov 09, 2017 6:34:39 PM org.jenkinsci.plugins.github.webhook.subscriber.PingGHEventSubscriber onEvent
INFO: PING webhook received from repo <https://github.example.com/example-org/example-repo>!

Push Event

A similar output to this is expected:

INFO: Received POST for https://github.example.com/example-org/repo-example
Nov 09, 2017 6:48:25 PM org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber$1 run
INFO: Poked jobExample
Nov 09, 2017 6:48:26 PM com.cloudbees.jenkins.GitHubPushTrigger$1 run
INFO: SCM changes detected in jobExample. Triggering #2

Pull Request Event

A similar output to this is expected:

Nov 09, 2017 8:22:04 PM com.cloudbees.jenkins.plugins.github_pull.PullRequestHookReceiver doIndex
INFO: Accepted GitHub pull request hook delivery 44b54ed0-c583-11e7-87f4-6814e09a595a
Nov 09, 2017 8:22:04 PM com.cloudbees.jenkins.plugins.github_pull.PullRequestPayload call
INFO: Processing hook call for pull-request #9 for repo-example
  remote: Counting objects
  remote: Compressing objects
  Receiving objects
    Done: 57
  Resolving deltas
    Done: 26
  Updating references
    Done: 1
Nov 09, 2017 8:22:07 PM com.cloudbees.jenkins.plugins.github_pull.PullRequestPayload call
INFO: Added validated merge to GitHub pull request #9 to example-org/repo-example
Nov 09, 2017 8:22:23 PM hudson.model.Run execute
INFO: jobExample #6 main build action completed: SUCCESS

Monitoring your API quota

If you see intermittent issues, and you are not sure if your GitHub API quota is getting exhausted at times, first check the controller logs for Jenkins-Imposed API Limiter: messages, as those should appear when rate limiting is happening.

If you don’t have monitoring to check the API quota directly, you can instead configure a new job in the folder that contains your credential being used, with the following Pipeline code to call Get rate limit status for the authenticated user:

pipeline {
    agent { label 'my-github-agent' }
    stages {
        stage('test api quota') {
            steps {
                withCredentials([usernamePassword(credentialsId: 'CREDENTIAL_ID', passwordVariable: 'TOKEN', usernameVariable: 'USER')]) {
                    sh '''curl -L \
                          -H "Accept: application/vnd.github+json" \
                          -H "Authorization: Bearer $TOKEN" \
                          -H "X-GitHub-Api-Version: 2022-11-28" \
                          https://api.github.com/rate_limit \
                          | jq .rate
                    '''
                }
            }
        }
    }
}▼

The result of this job will show your rate limit:

[Pipeline] sh
+ + curl -L -H Accept: application/vnd.github+json -H Authorization: Bearer **** -H X-GitHub-Api-Version: 2022-11-28 https://api.github.com/rate_limit
jq .rate
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  1422  100  1422    0     0   4286      0 --:--:-- --:--:-- --:--:--  4296
{
  "limit": 5000,
  "used": 8,
  "remaining": 4992,
  "reset": 1692210725
}

Please be warned that this exposes the API token on the process listing (ps -ef) of the agent used (though it will be redacted from your Pipeline build logs), so it’s important to only run this on a trusted agent that does not have other users who can access it, and the commands run on this machine are not logged and uploaded to an external service.

If you configure this job to build periodically with a cron trigger, you can then graph the remaining output versus the time that the build ran, and see a visual overview of your remaining rate limit over time (to determine if there are certain periods of the day, or the week, where the rate limit is being exhausted).

The Restrict GitHub API requests only when near or above rate limit (ThrottleOnOver) strategy will throttle GitHub API requests when there are 15 remaining, or your limit divided by 20 remaining, whichever is greater, as of github-branch-source-plugin 1732.v3f1889a_c475b_.

Tested products/plugins version

The latest update of this article has been tested with:

GitHub plugin 1.27.0
GitHub Branch Source Plugin 2.3.1
CloudBees Pull Request Builder for GitHub 1.11
GitHub API v3