Bitbucket Webhooks Troubleshooting

Symptoms

Jobs are not automatically triggered when a changes are pushed to Bitbucket

Diagnostic/Treatment

Pre-condition: Jenkins (resp. Bitbucket) can reach out to Bitbucket (resp. Jenkins)

Non-Multibranch

In Jenkins for non-multibranch jobs, you are either using the Git Plugin, Bitbucket Plugin or the Bitbucket Push and Pull Request Plugin.

Git Plugin

1) Endpoint

This applies to repository configured with a hook to the endpoint $JENKINS_URL/git/notifyCommit

2) Add-on (for Bitbucket Server)

This endpoint is used by the add-on Webhook to Jenkins for Bitbucket in Bitbucket Server:

3) Review Configuration

Review the articles:

4) Troubleshooting

To troubleshoot request processing on that endpoint, please add a logger to the following class:

hudson.plugins.git.GitStatus

Expected Logs

With branch and/or sha specified:

2017-03-10 08:07:09.273+0000 [id=831118]    FINE    h.p.g.GitStatus$JenkinsAbstractProjectListener#onNotifyCommit: Received notification for uri = https://my.bitbucket.server/myuser/myrepo.git ; sha1 = e3baa50a43479d6d0246fa2ba5d921c0e06fc707 ; branches = [develop]
2017-03-10 08:07:09.273+0000 [id=831118]    FINE    h.p.g.GitStatus$JenkinsAbstractProjectListener#onNotifyCommit: Branch Spec origin/develop matches modified branch develop for mybitbucket » myrepo.
2017-03-10 08:07:09.273+0000 [id=831118]    INFO    h.p.g.GitStatus$JenkinsAbstractProjectListener#onNotifyCommit: Scheduling folder-bitbucket » myrepo to build commit e3baa50a43479d6d0246fa2ba5d921c0e06fc707

Without branch and/or sha specified:

2017-03-13 02:39:51.532+0000 [id=106781]	FINE	h.p.g.GitStatus$JenkinsAbstractProjectListener#onNotifyCommit: Received notification for uri = https://my.bitbucket.server/myuser/myrepo.git ; sha1 = null ; branches = []
2017-03-13 02:39:51.532+0000 [id=106781]	INFO	h.p.g.GitStatus$JenkinsAbstractProjectListener#onNotifyCommit: Triggering the polling of mybitbucket » myrepo

Bitbucket Plugin and Bitbucket Push and Pull Request Plugin

Bitbucket Push And Pull Request Plugin will not work if the Bitbucket Plugin is installed.

1) Endpoint

This applies to Bitbucket repositories configured with a hook to the endpoint $JENKINS_URL/bitbucket-hook/.

2) Add-on (for Bitbucket Server)

The Bitbucket Plugin functionality was broken prior to version 1.1.7 of the plugin. You can find more details in JENKINS-28877.

Bitbucket Plugin integrates with Bitbucket Server via the add-on Bitbucket Server Web Post Hooks Plugin in Bitbucket server:

post service webhook for bitbucket server

3) Review Configuration

Review the articles:

4) Troubleshooting

To troubleshoot request processing on that endpoint, please add a logger to the following class:

com.cloudbees.jenkins.plugins.BitbucketJobProbe com.cloudbees.jenkins.plugins.BitBucketTrigger com.cloudbees.jenkins.plugins.BitbucketPayloadProcessor com.cloudbees.jenkins.plugins.BitbucketHookReceiver io.jenkins.plugins.bitbucketpushandpullrequest

Expected Logs (Bitbucket Plugin)

2017-03-13 02:32:36.138+0000 [id=127609]	FINE	c.c.j.p.BitbucketHookReceiver#doIndex: Received commit hook notification : {<payload>}
2017-03-13 02:32:36.207+0000 [id=127609]	INFO	c.c.j.p.BitbucketPayloadProcessor#processPayload: Processing new Webhooks payload
2017-03-13 02:32:36.208+0000 [id=127609]	INFO	c.c.j.p.BitbucketPayloadProcessor#processWebhookPayload: Received commit hook notification for {<simplified-payload>}
2017-03-13 02:32:36.307+0000 [id=127609]	FINE	c.c.j.plugins.BitbucketJobProbe#triggerMatchingJobs: Considering candidate job freestyle-push
2017-03-13 02:32:36.307+0000 [id=127609]	FINE	c.c.j.plugins.BitbucketJobProbe#triggerMatchingJobs: Considering to poke support-team » Bucket » CE-424 » BitbucketCloud » freestyle-push
2017-03-13 02:32:36.311+0000 [id=127609]	INFO	c.c.j.plugins.BitbucketJobProbe#triggerMatchingJobs: Triggering BitBucket job freestyle-push

Expected Logs (Bitbucket Push and Pull Request Plugin)

2021-09-10 09:44:11.803+0000 [id=231613]	INFO	i.j.p.b.BitBucketPPRHookReceiver#doIndex: Received x-event-key: pr:opened from Bitbucket
2021-09-10 09:44:11.807+0000 [id=231613]	INFO	i.j.p.b.p.BitBucketPPRPayloadProcessorFactory#createProcessor: Create BitBucketPPRPullRequestServerPayloadProcessor
2021-09-10 09:44:11.807+0000 [id=231613]	INFO	i.j.p.b.o.BitBucketPPRObserverFactory#createObservable: Add BitBucketPPRPullRequestServerObserver for BitBucketPPREvent [event=pr, action=opened]
2021-09-10 09:44:11.808+0000 [id=231613]	INFO	i.j.p.b.BitBucketPPRJobProbe#matchGitScm: Trying to match {<repository>}<-->{<repository>}
2021-09-10 09:44:11.808+0000 [id=231613]	INFO	i.j.p.b.BitBucketPPRJobProbe#matchGitScm: Matched scm
2021-09-10 09:44:11.808+0000 [id=231613]	INFO	i.j.p.b.f.BitBucketPPRFilterMatcher#getMatchingFilters: Event BitBucketPPREvent [event=pr, action=opened] matches trigger filter io.jenkins.plugins.bitbucketpushandpullrequest.filter.pullrequest.server.BitBucketPPRPullRequestServerTriggerFilter@32e62fd5
2021-09-10 09:44:11.811+0000 [id=231613]	INFO	i.j.p.b.BitBucketPPRHookReceiver#doIndex: Response sent.
2021-09-10 09:44:12.566+0000 [id=237649]	INFO	i.j.p.b.BitBucketPPRTrigger$1#onPollSuccess: The polling of the BB PPR job was successful for the cause: BitBucketPPRTriggerCause [bitbucketAction=BitBucketPPRPullRequestServerAction]
2021-09-10 09:44:12.566+0000 [id=237648]	INFO	i.j.p.b.util.BitBucketPPRUtils#lambda$matches$2: Matched branch: {<branch>}
2021-09-10 09:44:12.566+0000 [id=237649]	INFO	i.j.p.b.BitBucketPPRTrigger#scheduleJob: Check if job should be triggered due to changes in SCM
2021-09-10 09:44:12.569+0000 [id=237649]	INFO	i.j.p.b.BitBucketPPRTrigger#scheduleJob: SCM changes detected in bitbucket. Triggering  #{<build number>}
2021-09-10 09:44:19.040+0000 [id=237649]	INFO	i.j.p.b.o.BitBucketPPRObservable#notifyObservers: Event: BitBucketPPRBuildStarted {<simplified-payload>}
2021-09-10 09:44:23.567+0000 [id=237646]	INFO	o.j.h.t.v.VersionNumberCommon#getPreviousBuildWithVersionNumber: Previous build's version-number: {<build version number>}
2021-09-10 09:44:23.567+0000 [id=237646]	INFO	o.j.h.t.v.VersionNumberCommon#formatVersionNumber: Version-number format-string before expansion of env-variables: '${BUILD_YEAR}.${BUILD_MONTH}.${BUILDS_THIS_MONTH}'
2021-09-10 09:44:24.053+0000 [id=237648]	INFO	i.j.p.b.o.BitBucketPPRObservable#notifyObservers: Event: BitBucketPPRBuildStarted [context=BitBucketPPREventContext {<simplified-payload>}
2021-09-10 09:44:24.297+0000 [id=237649]	INFO	i.j.p.b.o.BitBucketPPRObservable#notifyObservers: Event: BitBucketPPRBuildFinished [context=BitBucketPPREventContext {<simplified-payload>}

Multibranch

In Jenkins, for multibranch jobs, you should be using the Bitbucket Branch Source Plugin.

1) Endpoint

This applies to repositories configured with a hook to the endpoint:

$JENKINS_URL/bitbucket-scmsource-hook/notify?server_url=<BITBUCKET_URL> for the native webhooks in Bitbucket
$JENKINS_URL/bitbucket-scmsource-hook/notify in case you are using Bitbucket WebPost Hooks Plugin
An example of endpoint for the native webhooks is -> https://jenkins.example.com/teams-dse-emea/bitbucket-scmsource-hook/notify?server_url=https%3A%2F%2Fbitbucket.example.com

2) Add-on (for Bitbucket Server)

Bitbucket Branch Source Plugin integrates with Bitbucket Server via the add-on Post Webhooks for Bitbucket in Bitbucket server:

3) Review Configuration

Review the articles:

4) Troubleshooting

To troubleshoot request processing on that endpoint, please add a logger to the following class:

com.cloudbees.jenkins.plugins.bitbucket.hooks

Expected Logs

Registering Hooks:

2017-03-12 23:50:10.659+0000 [id=28857]	INFO	c.c.j.p.b.h.WebhookAutoRegisterListener#registerHooks: Registering hook for jenkinssupport/simple-project
2017-03-13 00:15:57.702+0000 [id=28857]	FINE	c.c.j.p.b.h.WebhookAutoRegisterListener#removeHooks: NOT removing hook for jenkinssupport/simple-project because does not exists or its used in other project

Processing Reindexing:

2017-03-13 02:33:27.697+0000 [id=132556]	INFO	c.c.j.p.b.h.PushHookProcessor#process: Received hook from Bitbucket. Processing push event on jenkinssupport/simple-project
2017-03-13 02:33:27.699+0000 [id=132556]	INFO	c.c.j.p.b.hooks.HookProcessor$1#run: Multibranch project found, reindexing jenkinssupport
2017-03-13 02:33:27.699+0000 [id=132556]	INFO	c.c.j.p.b.hooks.HookProcessor$1#run: Multibranch project found, reindexing simple-project
2017-03-13 02:33:27.700+0000 [id=132556]	INFO	c.c.j.p.b.hooks.HookProcessor$1#run: Multibranch project found, reindexing pipeline-multibranch

5) Indexing Logs

Any event received on this endpoint should trigger a Branch indexing. But only the logs of the last branch indexing are persisted. If an issue seems intermittent and it is difficult to catch the Branch Indexing that exposes it. In order to catch the issue, you can set the system property com.cloudbees.hudson.plugins.folder.computed.FolderComputation.BACKUP_LOG_COUNT to for example 10 in order to keep the logs of the last 10 branch indexing.

You can set this property on startup by adding it to Jenkins Java arguments or run the following Groovy script in Manage Jenkins > Script Console:

com.cloudbees.hudson.plugins.folder.computed.FolderComputation.BACKUP_LOG_COUNT=10

The branch indexing files are located in the File System in $JENKINS_HOME/jobs/<pathToMultibranch>/indexing/ The folder computation files are located in the File System in $JENKINS_HOME/jobs/<pathToMultibranch>/computation/

Troubleshooting Further

If for some reason the request is not received by Jenkins and improved logging did not help, you can troubleshoot this even further by enabling Access Logging for Jenkins.

Expected Logs

Bitbucket Cloud

192.168.0.1 - - [13/Mar/2017:13:23:28 +1000] "POST /bitbucket-scmsource-hook/notify HTTP/1.1" 200 0 "-" "Bitbucket-Webhooks/2.0"
192.168.0.1 - - [13/Mar/2017:13:23:28 +1000] "POST /bitbucket-hook/ HTTP/1.1" 200 0 "-" "Bitbucket-Webhooks/2.0"

Bitbucket Server

192.168.0.2 - - [13/Mar/2017:13:23:52 +1000] "POST /bitbucket-scmsource-hook/notify HTTP/1.1" 200 0 "-" "Atlassian HttpClient 0.23.0 / Bitbucket-4.13.0 (4013000) / Default"
192.168.0.2 - - [13/Mar/2017:13:23:53 +1000] "GET /git/notifyCommit HTTP/1.1" 200 110 "-" "Apache-HttpClient/4.5.1 (Java/1.8.0_102)"
192.168.0.2 - - [13/Mar/2017:13:23:53 +1000] "POST /bitbucket-hook/ HTTP/1.1" 500 3348 "-" "Bitbucket 4.13.0"

Most common issues

The hook URL is not correctly configured. See JENKINS-55649 for more information.

On this case, by enabling the custom loggers previously explained on this case, you should get the stacktrace below on Jenkins once the webhook arrives to Jenkins. The problem is that the server_url parameter is not sent by Bitbucket.

X-Bitbucket-Type header / server_url request parameter not found. Bitbucket Cloud webhook incoming.

Recall that the endpoint definition is not exactly the same for the native webhooks and the Bitbucket WebPost Hooks Plugin.

Native webhooks in Bitbucket endpoint definition: $JENKINS_URL/bitbucket-scmsource-hook/notify?server_url=<BITBUCKET_URL>
Bitbucket WebPost Hooks Plugin endpoint definition: $JENKINS_URL/bitbucket-scmsource-hook/notify

An example of endpoint for the native webhooks is -> https://jenkins.example.com/teams-dse-emea/bitbucket-scmsource-hook/notify?server_url=https%3A%2F%2Fbitbucket.example.com

References

Bitbucket Integration