Enable external notification events with external HTTP endpoints

External HTTP endpoints work with Cross Team Collaboration to enable external systems such as GitHub or Nexus to generate notification events for Pipelines.

Jenkins external notification webhooks are user-defined HTTP callbacks that can be triggered by external actions, such as pushing a new artifact to a Nexus repository. When this happens, the external system makes an HTTP request to this endpoint and passes a data payload in the body of the request. Jenkins broadcasts the request via the internal messaging API to all jobs that are configured with an event trigger.

Webhook events are typically secured with a shared secret that the sender uses to generate a unique message digest. The message digest is sent to the receiving system in an HTTP request header. Jenkins uses the digest and the shared secret to verify that the message has not been altered and that it originates from a trusted source.

In addition to using a shared secret, some webhooks support the use of IP address allowlists, which allows an Administrator to specify a list of addresses (or address ranges) that are allowed to create a webhook events.

Validating the contents of the incoming message using a shared secret known only by the sender and the receiver ensures that rogue users are not able to send "fake" events which would cause Jenkins to trigger unexpected actions.

Figure 1. External HTTP Endpoints overview

The external HTTP endpoints feature requires the installation of the following plugins on the operations center instance:

Notification API (required): responsible for sending the messages across teams and jobs.
Operations Center Notification: provides the router to transfer the messages across different teams
External Notification Plugin: management screen for webhooks

Configuring

External webhook endpoints are added and deleted from Manage Jenkins Manage Notification Webhook HTTP Endpoints.

Figure 2. The external HTTP endpoints configuration page

The Manage Notification Webhook HTTP Endpoints option will only appear if the external-notification-plugin plugin is installed.

The configuration page also shows the details of current endpoints.

Figure 3. The external HTTP endpoints configuration page

The configuration screen shows all currently enabled endpoints, with the following details:

Name - The user friendly name given to this endpoint
Endpoint URL - The URL by which this endpoint can be accessed
Last Event - The date/time at which this endpoint was last accessed
Last Status - The HTTP status code associated with the last request received by this endpoint
Webhook Message Format - The type of message that this endpoint accepts
Details - Displays the full details for this endpoint
Delete - Deletes this endpoint.

Note: This action cannot be undone.

Hovering over the Last Status field or clicking on the Details button shows the HTTP status message associated with the last request received by an endpoint. This can be very helpful when troubleshooting new webhook configurations.

External HTTP notification endpoint types

Several endpoint types are already included. Custom endpoints can be defined by specifying the security type and relevant details. The following endpoint types are currently supported:

Custom JSON Webhook
Bitbucket Cloud
Bitbucket Server
Docker Hub
GitHub
Nexus 3

To configure a new endpoint, click Register a new endpoint.

Custom JSON webhooks

The Custom JSON webhook allows full control of the configuration of a new webhook endpoint. This allows integrations with systems not supported with an out of the box integration. Custom JSON webhooks support two forms of security - HMAC message signatures and IP address allowlists. At least one of these security types must be chosen, and if desired, both can be used together.

Custom JSON webhooks require the use of at least one type of security - either HMAC signatures or IP address allowlists.

Custom JSON webhooks using HMAC signature validation

The custom JSON webhook allows full control of the configuration of a new webhook endpoint. This allows integrations with systems not supported with an out of the box configuration.

Figure 4. Configuring a custom JSON webhook endpoint with HMAC signature validation

Webhook Name - A user friendly name for this new endpoint
HMAC HTTP Header - The name of the HTTP request header that will contain the message digest
Message Signature Validation Type - The cryptographic hash type that will be used to generate the message digest
Secret - The shared secret that will be used to validate the authenticity of received events. Use the Generate Secret button to automatically generate a cryptographically random secret.

Refer to HMAC (hash-based message authentication code) for more information on this widely used method of verifying the integrity and authenticity of a message.

Custom JSON webhook with allowlists

Not all webhook providers support HMAC signature validation. For those that don’t, it is possible to provide a level of security by allowlisting the IP addresses of the system that will generate the webhook events. To do this, check the box labeled "Define a list of IP addresses that are allowed to send events to this endpoint" and enter the IP addresses (one per line) of the remote host which will be generating the webhook events. You may enter both IPv4 and IPv6 addresses, as well as address ranges using CIDR notation.

Figure 5. Configuring a custom JSON webhook endpoint with an IP allowlist

When choosing NONE as the Message Signature Validation Type the allowlist entry will automatically be shown. Although it is not recommended, you may enter 0.0.0.0/0 and ::/0 which will allow access from any host. If no addresses are specified, then access from any host will be denied.

Bitbucket Cloud

The Bitbucket Cloud webhook endpoint is pre-configured to work with Bitbucket Cloud events. Bitbucket Cloud does not support HMAC signatures, and per Atlassian’s documentation, it is recommended to secure your endpoint using an IP allowlist. The Bitbucket Cloud webhook is pre-configured with the IP addresses from Atlassian’s documentation.

Configuring a Bitbucket Cloud webhook endpoint

The following fields are required to create a new Bitbucket Cloud endpoint:

Webhook Name - A user friendly name for this new endpoint
Allowlist - A list of IP addresses (or address ranges) specifying the hosts that are allowed to send Bitbucket Cloud webhook events to your Jenkins instance.
Secret - (Optional) A unique string which will be use to validate the authenticity of incoming webhook events. Use the Generate Secret button to automatically generate a cryptographically random secret, or leave this field blank if you do not wish to perform this validation (this is not recommended).

It is also possible to specify a shared secret which will be included in the URL that is generated for a Bitbucket Cloud webhook endpoint. Although no signature validation will be performed, incoming events that do not include this secret URL parameter will be ignored. Be sure to copy the entire url shown in the Manage Notification Webhook Endpoints list to use when configuring your Bitbucket Cloud webhook.

Figure 6. Bitbucket Cloud Webhook URL With Secret

For more information about Bitbucket Cloud webhooks, please refer to Manage Webhooks documentation provided by Atlassian.

Bitbucket server

The Bitbucket Server webhook endpoint is pre-configured to work with Bitbucket Server events, and supports HMAC SHA256 message signatures.

Figure 7. Configuring a Bitbucket Server Webhook Endpoint

The following fields are required to create a new Bitbucket Server endpoint:

Webhook Name - A user friendly name for this new endpoint
Secret - The shared secret that will be used to validate the authenticity of received events. Use the Generate Secret button to automatically generate a cryptographically random secret.

Refer to the Bitbucket Server documentation for more infomation on Bitbucket Server webhooks.

Docker Hub

The Docker Hub webhook endpoint is pre-configured to work with Docker Hub webhook events. Docker Hub does not support HMAC message signatures, so it is highly recommended to specify a secret which will be included with every webhook request that Docker Hub makes to your Jenkins instance. When a secret is configured then only requests which include this will be accepted.

Figure 8. Configuring a Docker Hub Webhook Endpoint

The following fields are required to create a new Docker Hub endpoint:

Webhook Name - A user friendly name for this new endpoint
Secret - (Optional) A unique string which will be use to validate the authenticity of incoming webhook events. Use the Generate Secret button to automatically generate a cryptographically random secret, or leave this field blank if you do not wish to perform this validation (this is not recommended).

Figure 9. Docker Hub Webhook With Secret

Be sure to copy the entire url shown in the Manage Notification Webhook Endpoints list to use when configuring your Docker Hub webhook.

Nexus 3 webhook

The Nexus 3 webhook endpoint allows for integration with Nexus events. Nexus webhooks are secured with a shared secret and allow you to subscribe to certain events on the Nexus repository.

Figure 10. Configuring a Nexus 3 webhook endpoint

The following fields are required to create a new Nexus endpoint:

Webhook Name - A user friendly name for this new endpoint
Secret - The shared secret that will be used to validate the authenticity of received events. Use the Generate Secret button to automatically generate a cryptographically random secret.

Refer to the Nexus documentation for more information on Nexus webhooks.

GitHub webhook

The GitHub webhook endpoint integrates with GitHub events. GitHub webhooks are secured with a shared secret and allow you to subscribe to events from GitHub whenever certain actions, such as tag/label creation or GitHub release, occur.

Figure 11. Configuring a GitHub webhook endpoint

The following fields are required to create a new GitHub endpoint:

Webhook Name - A user friendly name for this new endpoint
Secret - The shared secret that will be used to validate the authenticity of received events. Use the 'Generate Secret' button to automatically generate a cryptographically random secret.

Refer to the GitHub documentation for more information on GitHub webhooks.

Integrating webhooks with pipeline event triggers

A webhook payload is used by a matching event trigger in the pipeline jobs that are interested in the message. Ensure that the following plugins are installed on the controller running the pipeline:

pipeline-event-step (required): provides a Pipeline step to add a new event publisher or trigger
operations-center-notification: provides the router to transfer the messages across different teams

Refer to the Cross Team Collaboration documentation to configure notifications.

When a webhook receives an event, all the Pipelines listening on events (event triggers) will be queried. The Pipelines with a trigger condition that matches the publishEvent will be executed. The trigger step uses query patterns based on jmespath syntax to match events. Triggers can be configured directly from the Pipeline job configuration page or can be written as a Pipeline step.

Example webhook event trigger

This GitHub webhook example assumes you have installed the required plugins and that you have an existing GitHub account.

1 - Create a New GitHub Webhook Endpoint

From Manage Jenkins Manage Notification Webhook Endpoints, create a new GitHub webhook with the following values:

Name - My First Webhook
Secret - NotAGoodSecret

Figure 12. Create a new webhook endpoint for GitHub

2 - Create a New Webhook on GitHub

Next, from the Settings tab on your GitHub repository, select Webhooks on the left hand side and click Add webhook. Fill in the form values as shown:

Payload URL - Paste the Jenkins webhook endpoint URL that was generated in the first step
Content type - Select application/json from the dropdown list
Secret - Enter the shared secret value (NotAGoodSecret) from the first step
The rest of the form values can be left at their defaults

After filling in the details, click Add webhook to save the changes.

Figure 13. Create a new webhook on GitHub

3 - Validate the New Webhook Endpoint

After the webhook is defined, GitHub will send a test message to your Jenkins endpoint. Click Edit for the webhook you created and scroll to the bottom of the page. If all went well, it will show something like:

Figure 14. Validating your new endpoint

If you see a red triangle instead of a green check mark, click on the message id to see more details. Make needed changes to your configuration and click on the Redeliver button to validate again.

4 - Create a Pipeline Job Triggered by the Webhook Event

Create a Pipeline that will trigger when an event is received. We will use the GitHub test message and create a job that will build when it sees the attribute "zen" with a value of "Non-blocking is better than blocking.". Create a new Pipeline job and copy the following script:

// Declarative //
pipeline {
    agent any

    triggers {
        eventTrigger jmespathQuery("zen=='Non-blocking is better than blocking.'")
    }

    stages {
        stage('Example') {
            steps {
                sh 'echo Hello World';
            }
        }
    }
}

// Script //
properties([pipelineTriggers([eventTrigger jmespathQuery("zen=='Non-blocking is better than blocking.'"))])])

node {
    stage('Example')  {
        echo 'Hello World'
    }
}▼

Save your job and then click Build Now once to ensure the event triggers are properly registered.

5 - Test!

Clicking Redeliver from the GitHub repository webhook configuration will send an event and broadcast the message to all listening jobs. The Pipeline job created in the previous step will see this event, and trigger a build!

[Pipeline] {
[Pipeline] stage
[Pipeline] { (Example)
[Pipeline] sh
[Github Trigger] Running shell script
+ echo Hello World
Hello World
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS▼

This is a very simple example illustrating the basic steps needed to integrate external webhook events with Jenkins. For more details about writing build triggers, please refer to Cross Team Collaboration

Troubleshooting

There are a number of steps that can be helpful when troubleshooting webhooks:

Endpoint "Last Event"/"Last Status" fields do not show any webhook request was received
- Verify the endpoint url is accessible outside the firewall
- Verify that the external system that is generating webhook events is configured with the correct URL for your endpoint
Verify that the external system is using the expected mime-type of application/json
- Click Details to view the error message associated with the most recent request
Endpoint receives a message, but the "Last Status" shows an error code
- Click Details to view the error message associated with the most recent request
Endpoint successfully receives a message, but build jobs are not triggering as expected
- Verify that Notification is enabled and configured
- Verify that your JMESPath trigger query is correct
- Create a new log recorder for com.cloudbees.opscenter.plugins.notification.endpoint.WebhookEndpoint and set the level to FINEST in order to view the content of the webhook message received by Jenkins. This can be helpful in creating and troubleshooting event trigger queries.

The Details view can be very helpful when troubleshooting webhook problems. In addition to allowing you to verify the endpoint configuration, the details view will show additional information about the type of error you may be encountering.

In August 2020, the Jenkins project voted to replace the term master with controller. We have taken a pragmatic approach to cleaning these up, ensuring the least amount of downstream impact as possible. CloudBees is committed to ensuring a culture and environment of inclusiveness and acceptance - this includes ensuring the changes are not just cosmetic ones, but pervasive. As this change happens, please note that the term master has been replaced through the latest versions of the CloudBees documentation with controller (as in managed controller, client controller, team controller) except when still used in the UI or in code.