Symptoms
-
Jobs are not automatically triggered when after a
Push
orPull request
event in GitHub (and/or GitHub Enterprise)
Diagnostic/Treatment
-
Pre-condition : Jenkins (resp. GitHub) can reach out to GitHub (resp. Jenkins)
This troubleshooting requires from analysis of GitHub which generates a payload and Jenkins which parses it.
A. On GitHub
A.1 Webhook has not been generated or it is disabled (red cross)
-
Check that "GitHub-API-User" has
Admin
permissions for the repo. OrOwner
role for Webhook at Organization level. -
If GitHub
Username and Password
is used for GitHub plugin configuration, validate that the user has not enabled 2FA. -
Important Notice GitHub is eliminating password-based API access (as mentioned in GitHub documentation), we strongly recommend you to switch your passwords so that you use the GitHub Access tokens or you use a different method to authenticate such as GitHub App authentication.
A.2 When the Webhook is generated
As explained in GitHub Testing Webhooks.
Click on Redeliver
and evaluate the response and see if the Jenkins receive the payload in B.3 GitHub Hooks Problems Console and B.4 Dedicated Logger.
B. On Jenkins
B.1 Regenerate all Webhooks for your Jenkins instance
Go to Manage Jenkins > Configure System > GitHub plugin > Advance > Re-registers hooks for all jobs.
B.2 Validate Plugin Configuration
-
Identify which GitHub Server ("GitHub Server X") contains the repository/organization where webhook is failing.
-
For that server ("GitHub Server X"), validate its "GitHub API User" and its Rate limits by Manage Jenkins > GitHub plugin > "GitHub Server X" >
Test Connection
Verify credentials Credentials verified for user "GitHub-API-User", rate limit: 4994
3.Validate the configuration of $JENKINS_HOME/github-plugin-configuration.xml
:
```xml <github-plugin-configuration plugin="github@1.29.4"> <configs> <github-server-config> <name>GITHUB</name> <apiUrl>https://api.github.com</apiUrl> <manageHooks>true</manageHooks> <credentialsId>GH-mock-ST</credentialsId> <clientCacheSize>20</clientCacheSize> </github-server-config> </configs> <hookSecretConfig> <credentialsId></credentialsId> </hookSecretConfig> ```
-
A parent
<github-server-config>
is need to each of the GitHub and GitHub Enterprise Servers if you would like the plugin to manage hooks for you. -
For each server, you need to have configured a credential Secret Text GitHub API Token:
<credentialsId>GH-example-APIToken-ST</credentialsId>
or GitHub App private key -
For each server, you need to have configured
<manageHooks>true</manageHooks>
to enableRe-registers hooks for all jobs
B.3 Validate Job Configuration
Review the plugin a jobs configurations as explained in GitHub WebHook Configuration for each of type of jobs.
B.4 GitHub Hooks Problems Console
Available from Manage Jenkins > GitHub Hooks Problems Console only and if only any of Automatic Webhooks has not been setup in GitHub.
This page shows problems with webhooks, and ignored projects.
Notes: Another warning messages also can be found at:
-
in Manage Jenkins
-
in the trigger configuration for For Non-Multibranch jobs
B.5 Check if the computed folder is at maximum indexing capacity
If you collect a support bundle with the Build queue
option checked, then look at the buildqueue.md
file in that support bundle, check if there is the string:
Why in queue: At maximum indexing capacity
This message is emitted by this code, when the indexing limit is reached.
To explain why this is relevant, it is because an Organization folder is a ComputedFolder, and ComputedFolders have an indexing limit, and if you have too many Organization folder items on a controller being processed concurrently (for example if you have a monolith controller), you can reach this limit, and job triggering for organization folders will be delayed.
The limit can be adjusted using this system property:
-Dcom.cloudbees.hudson.plugins.folder.computed.ThrottleComputationQueueTaskDispatcher.LIMIT=10
B.6 Dedicated Logger
Create a logger for GitHub in https://<JENKINS_URL>/log/
at the ALL
level for (depending on the plugins you are using):
-
org.jenkinsci.plugins.github_branch_source
For the GitHub Branch Source plugin (Multbranch and Organization folders from GitHub) -
com.cloudbees.jenkins.plugins.bitbucket
for the Bitbucket Branch Source plugin (Multbranch and Team/Project folders from Bitbucket) -
org.jenkinsci.plugins.github
andcom.cloudbees.jenkins.GitHubWebHook
for the GitHub plugin (webhooks sent to$JENKINS_URL/github-webhook/
) -
hudson.plugins.git.GitStatus
for the Git plugin GitStatus class
Test Credentials
A similar output to this is expected:
Jul 01, 2016 12:51:38 PM FINE org.jenkinsci.plugins.github.webhook.GHEventHeader$PayloadHandler parse Header X-GitHub-Event -> null Jul 01, 2016 12:51:41 PM FINE org.jenkinsci.plugins.github.internal.GitHubLoginFunction applyNullSafe Create new GH client with creds id 021643c7-ef89-4014-b835-6fab9add5165
Push payload
A similar output to this is expected:
Jul 01, 2016 12:32:03 PM FINE org.jenkinsci.plugins.github.webhook.GHEventHeader$PayloadHandler parse Header X-GitHub-Event -> push Jul 01, 2016 12:32:03 PM FINEST org.jenkinsci.plugins.github.webhook.GHEventPayload$PayloadHandler parse Payload {"ref":"refs/heads/master","before":"a1a91ab98386e4eced8ce0720d5c44158e85a3db","after":"58831ba23954684d98c91fdd303b2c8cebc7c504","created":false,"deleted":false,"forced":false,"base_ref":null,"compare":"https://github.com/example-org/example-repo/compare/a1a91ab98386...58831ba23954","commits": ... Jul 01, 2016 12:32:03 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent Received POST for https://github.com/example-org/example-repo Jul 01, 2016 12:32:03 PM FINE org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber$1 run Considering to poke ZD-37633-CBGitHubPRBuilder Jul 01, 2016 12:32:03 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber$1 run Poked ZD-37633-CBGitHubPRBuilder
Pull Request payload
A similar output to this is expected:
Jul 01, 2016 2:33:23 PM FINE org.jenkinsci.plugins.github.webhook.GHEventHeader$PayloadHandler parse Header X-GitHub-Event -> pull_request Jul 01, 2016 2:33:23 PM FINEST org.jenkinsci.plugins.github.webhook.GHEventPayload$PayloadHandler parse Payload {"action":"closed","number":2,"pull_request":{"url":"https://api.github.com/repos/example-org/example-repo/pulls/2","id":75972749,"html_url":"https://github.com/example-org/example-repo/pull/2","diff_url":"https://github.com/example-org/example-repo/pull/2.diff","patch_url":"https://github.com/example-org/example-repo- ...
B.6 Jenkins logs
The following traces illustrate successful build triggers on Jenkins logs:
Webhook register
Nov 09, 2017 6:18:43 PM org.jenkinsci.plugins.github.config.GitHubPluginConfig doReRegister INFO: Called registerHooks() for 1 jobs Nov 09, 2017 6:18:43 PM org.jenkinsci.plugins.github.webhook.WebhookManager$1 run INFO: GitHub webhooks activated for job myJobExample with [GitHubRepositoryName[host=github.com,username=example-org,repository=example-repo]] (events: [PUSH, REPOSITORY])
In case the webhook was sucessfully, Jenkins receives a PING event from repo webhook so a similar trace like the following would appear:
Nov 09, 2017 6:34:39 PM org.jenkinsci.plugins.github.webhook.subscriber.PingGHEventSubscriber onEvent INFO: PING webhook received from repo <https://github.example.com/example-org/example-repo>!
Push Event
A similar output to this is expected:
INFO: Received POST for https://github.example.com/example-org/repo-example Nov 09, 2017 6:48:25 PM org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber$1 run INFO: Poked jobExample Nov 09, 2017 6:48:26 PM com.cloudbees.jenkins.GitHubPushTrigger$1 run INFO: SCM changes detected in jobExample. Triggering #2
Pull Request Event
A similar output to this is expected:
Nov 09, 2017 8:22:04 PM com.cloudbees.jenkins.plugins.github_pull.PullRequestHookReceiver doIndex INFO: Accepted GitHub pull request hook delivery 44b54ed0-c583-11e7-87f4-6814e09a595a Nov 09, 2017 8:22:04 PM com.cloudbees.jenkins.plugins.github_pull.PullRequestPayload call INFO: Processing hook call for pull-request #9 for repo-example remote: Counting objects remote: Compressing objects Receiving objects Done: 57 Resolving deltas Done: 26 Updating references Done: 1 Nov 09, 2017 8:22:07 PM com.cloudbees.jenkins.plugins.github_pull.PullRequestPayload call INFO: Added validated merge to GitHub pull request #9 to example-org/repo-example Nov 09, 2017 8:22:23 PM hudson.model.Run execute INFO: jobExample #6 main build action completed: SUCCESS
Monitoring your API quota
If you see intermittent issues, and you are not sure if your GitHub API quota is getting exhausted at times, first check the controller logs for Jenkins-Imposed API Limiter:
messages, as those should appear when rate limiting is happening.
If you don’t have monitoring to check the API quota directly, you can instead configure a new job in the folder that contains your credential being used, with the following Pipeline code to call Get rate limit status for the authenticated user:
pipeline { agent { label 'my-github-agent' } stages { stage('test api quota') { steps { withCredentials([usernamePassword(credentialsId: 'CREDENTIAL_ID', passwordVariable: 'TOKEN', usernameVariable: 'USER')]) { sh '''curl -L \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer $TOKEN" \ -H "X-GitHub-Api-Version: 2022-11-28" \ https://api.github.com/rate_limit \ | jq .rate ''' } } } } }
The result of this job will show your rate limit:
[Pipeline] sh + + curl -L -H Accept: application/vnd.github+json -H Authorization: Bearer **** -H X-GitHub-Api-Version: 2022-11-28 https://api.github.com/rate_limit jq .rate % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 1422 100 1422 0 0 4286 0 --:--:-- --:--:-- --:--:-- 4296 { "limit": 5000, "used": 8, "remaining": 4992, "reset": 1692210725 }
Please be warned that this exposes the API token on the process listing (ps -ef
) of the agent used (though it will be redacted from your Pipeline build logs), so it’s important to only run this on a trusted agent that does not have other users who can access it, and the commands run on this machine are not logged and uploaded to an external service.
If you configure this job to build periodically with a cron trigger, you can then graph the remaining
output versus the time that the build ran, and see a visual overview of your remaining rate limit over time (to determine if there are certain periods of the day, or the week, where the rate limit is being exhausted).
The Restrict GitHub API requests only when near or above rate limit (ThrottleOnOver ) strategy will throttle GitHub API requests when there are 15 remaining, or your limit divided by 20 remaining, whichever is greater, as of
github-branch-source-plugin 1732.v3f1889a_c475b_.
|