Cluster-wide job triggers

Depending on the scheme used to organize an operations center cluster, it may be necessary to trigger jobs that are on a remote client controller. For example, the team responsible for the production servers may want to trigger the QA team’s sanity tests against the staging environment servers before deploying to production. Given that the security concerns of the production servers can differ from those of the QA team’s, it may be that the QA team uses a different client controller from the production team. In order to solve these types of problems the Operations Center Context plugin provides a build step and a post build action that can trigger jobs either on the root operations center or any client controller Jenkins instance in the operations center cluster.

The ability to trigger jobs across controllers within an operations center cluster has been added to operations center as an update to the 1.6 release. This functionality requires the following minimum plugin versions:

Client controllers must be running:
- Operations Center Context plugin 1.6.1 or later
- Operations Center Client plugin 1.6.1 or later
Operations center server must be running:
- Operations Center Context plugin 1.6.1 or later
- Operations Center Server plugin 1.6.4 or later
- Operations Center Single Sign-On plugin 1.6.1 or later

If the operations center server has not been upgraded, client controllers will be unable to trigger jobs on remote controllers.

If individual client controllers have not been upgraded:

Those client controllers will be unable to trigger jobs on remote controllers.
Other remote controllers will be able to trigger jobs on the client controllers with Operations Center Context plugin 1.6, but the blocking behavior and parameterized trigger functionality will be silently unavailable for jobs running on the client controllers with Operations Center Context plugin 1.6.

The ability to trigger remote jobs is only possible with an adequate level of trust inside the operations center cluster. It is important to understand the Authentication mapping configuration between operations center and the client controllers.

Trigger builds on remote/local jobs build step

The Trigger builds on remote/local jobs build step is available for Freestyle jobs.

The Trigger builds on remote/local jobs build step allows you to intersperse triggering jobs with other build steps. This can be useful if there is a flow of operations needed by the job. For example, you may have a job that resets test database instances. Using this build step, you can trigger the job, specify the target database as a build parameter, wait for the reset job to complete, and then proceed with the build steps that actually run the tests against the test database.

To add the trigger to a Freestyle job:

Navigate to your Freestyle job.
In the left navigation menu, select Configure.
In the left navigation menu, select Build Steps.
Select +Add build step, and from the dropdown select Trigger builds on remote/local jobs to add the build step to trigger jobs across the operations center cluster.

You can trigger multiple downstream jobs from the same build step. To force sequential triggers, use multiple build steps.
Select Add downstream job to select the downstream job to trigger.
Configure the downstream job options.
Select Save to save the configuration.

All jobs in the same Trigger builds on remote/local jobs build step are processed in parallel. If you have a build step with two jobs where the first job is configured to wait until finished and mark the build as a failure, and the second job is configured to trigger only when successful, both jobs are triggered even if the first downstream job fails, because triggering occurs before results are known. However, if a job is deleted, the On job missing configuration may mark the build as FAILED and prevent remaining jobs from being triggered.

Configure the downstream job options

Configure the following options for each downstream job:

Name - enter the job path in the Name field using one of the following URI formats.

Type / after each path component to fetch the next level of completions.
- jenkins://./path/on/this/server: Local paths on the same instance.
- cjp:///path/on/operations/center: Paths via the operations center (more legible, but assumes controllers are not moved within operations center).
- jenkins://instance-id/path/on/another/controller: Paths on another controller using the instance ID (most stable, but not human-readable).
- cjp:///path/to/controller-item/path/on/another/controller: Cross-controller paths via the operations center.
Trigger When - allows you to choose, based on the current build state, when the trigger should run:

Figure 1. The available Trigger When options
- Always will trigger the selected job irrespective of the current build result
- Either unstable or successful will trigger the selected job if the current build has been marked as UNSTABLE or retains the initial build result of SUCCESS but will not trigger the selected job if a previous build step has marked the build as FAILED.
- Only when successful will trigger the selected job only if the current build retains the initial build result of SUCCESS. If a previous build step marked the build as either UNSTABLE or FAILED then the job will not be triggered.
On job missing - this allows you to choose what should happen to the current build if the specified job cannot be located (i.e. if it has been moved / deleted)

Mode - this allows specifying any blocking behavior for the triggered job.

Figure 2. The available Mode options

Fire and forget does not wait and assumes that the job will be triggered. The build request will be sent with a 24h time-to-live, so as long as the Jenkins instance sending the trigger connects to the operations center server (in order for the request to be forwarded to the operations center server messaging transport) and the target Jenkins instance subsequently connects to the operations center server within that 24h window the job will be triggered.

Figure 3. Fire and forget mode options
Wait until scheduled will wait, for a user configurable time period, for confirmation of the job being enqueued on the target Jenkins build queue. The build request will be sent with a time-to-live of the specified timeout. For example, if you specify a five-minute timeout, the build request will expire after 5 minutes; if you specify a 20-day timeout, then the build request will expire after 20 days.

If the confirmation is not received within the timeout, then the On timeout action will be applied to the build.

Multiple requests to build the downstream job using this mode can be coalesced by the downstream Jenkins.

Figure 4. Wait until scheduled mode options
Wait until started will wait, for a user configurable time period, for confirmation that the target job has started building. The build request will be sent with a time-to-live of the specified timeout. For example, if you specify a five-minute timeout, the build request will expire after 5 minutes; if you specify a 20-day timeout, then the build request will expire after 20 days.

If the confirmation is not received within the timeout, then the On timeout action will be applied to the build.

Once confirmation is received the downstream build number will be recorded on the build log.

Multiple requests to build the downstream job using this mode can be coalesced by the downstream Jenkins.

Figure 5. Wait until started mode options
Wait until finished will wait, for a user configurable time period, for confirmation that the target job has completed building. The build request will be sent with a time-to-live of 24h.

If the build result is not received within the timeout, then the On timeout action will be applied to the build.

When the build result is received, the downstream build number and the build result will be recorded on the build log.

If the build result is UNSTABLE then the On unstable action will be applied to the build.

If the build result is FAILED then the On failure action will be applied to the build.

Multiple requests to build the downstream job using this mode are not coalesced by the downstream Jenkins as the requirement to report back the tracking information ensures separate builds for each request.

Figure 6. Wait until finished mode options

Track progress and wait until finished allows complete control over monitoring the progression of the build. The Scheduled timeout controls how long to wait for the confirmation that the build is enqueued. This timeout also controls the time-to-live of the build request. If left blank then a 24h time-to-live will be used. In the event of the scheduled timeout expiring, the On scheduled timeout action will be applied to the build.

The Started timeout controls how long to wait for confirmation of the downstream job starting. If confirmation is not received within the specified timeout then the On started timeout action will be applied to the build. If confirmation is received then the downstream build number will be recorded on the build log.

Finally, the Finished timeout controls how long to wait for confirmation that the target job has completed building.

If the build result is not received within the timeout, then the On finished timeout action will be applied to the build.

When the build result is received, the downstream build number and the build result will be recorded on the build log.

If the build result is UNSTABLE then the On unstable action will be applied to the build.

If the build result is FAILED then the On failure action will be applied to the build.

Multiple requests to build the downstream job using this mode are not coalesced by the downstream Jenkins as the requirement to report back the tracking information ensures separate builds for each request.

The Track progress and wait until finished mode provides real-time state details about the downstream job or jobs being triggered, including a clickable job path and build number that provide a convenient way to see job details. Refer to the example below.

To include this output in your build logs, you must specify a timeout value for each of the job status states you want to track.

starting remote job
[Pipeline] triggerRemoteJob
[operations-center-context] Triggering Controller1 » localTarget

[operations-center-context] Waiting for triggered job
[operations-center-context]    Controller1 » localTarget
[operations-center-context] to be SCHEDULED...
[operations-center-context]    Controller1 » localTarget SCHEDULED
[operations-center-context] Finished waiting for triggered job(s) to be SCHEDULED after 0 ms

[operations-center-context] Waiting for triggered job
[operations-center-context]    Controller1 » localTarget
[operations-center-context] to be STARTED...
[operations-center-context]    Controller1 » localTarget #16 STARTED
[operations-center-context] Finished waiting for triggered job(s) to be STARTED after 9 sec

[operations-center-context] Waiting for triggered job
[operations-center-context]    Controller1 » localTarget
[operations-center-context] to be FINISHED...
[operations-center-context]    Controller1 » localTarget #16 SUCCESS
[operations-center-context] Finished waiting for triggered job(s) to be FINISHED after 1.7 sec▼

Figure 7. Track progress and wait until finished mode options

Each of the On … actions has four options:

Ignore and continue - which will leave the build result as is and continue.
Mark build as failure and continue - which will set the build result to FAILURE and continue.
Mark build as failure and stop - which will set the build result to FAILURE and try to stop the build without running any further build steps.
Mark build as unstable and continue - which will set the build result to UNSTABLE - unless it is already marked as FAILURE - and continue.

Figure 8. Configuring failure modes

Quiet period - an optional alternate quiet period (in seconds) before building. If unset, defaults to the quiet period defined by the downstream project, or the system-wide default quiet period.

Trigger parameterized jobs

In addition to the four mandatory configuration options for each downstream job you also have the option of specifying the build parameters that the job will be supplied with when building.

Parameters on a trigger for a non-parameterized downstream job will be stripped from the build request on receipt by the downstream Jenkins.

Similarly, any parameters that are not defined in a parameterized downstream job will be stripped from the build request.

This is to ensure that a trigger cannot maliciously manipulate the downstream build job’s environment as most build parameters get exposed as environment variables in the build run.

The Add parameters button can be used to add parameter value factories to the build request(s) for the downstream job. The parameter value factories are an extension point.

Figure 9. Configure downstream job parameters

Figure 10. A configured downstream job parameter

The Operations Center Context plugin provides the following parameter value factories:

Boolean parameter produces a single parameter value which can be either true or false;

Figure 11. Configuration options for a Boolean parameter
Current build parameters produces a single set of parameters that are a subset of the triggering job’s build parameters. This allows you to propagate the triggering job’s build parameters to the triggered job.

By default, parameters which are defined as having sensitive values - such as password parameters - will be excluded but the Parameters with sensitive values option allows this to be configured.

If you need to exclude some of the build parameters from the trigger use the Excluded parameters option. This takes a multi-line list of parameter names (one per line) which will be excluded from the build request. Wildcards matching is supported by using the * character.

Figure 12. Configuration options for Current build parameters
String parameter produces a single parameter value specified as a verbatim string constant.

Figure 13. Configuration options for a String parameter
Fan-out string parameter produces multiple sets of parameter values. This will result in multiple build requests of the downstream job. Each value is specified on a separate line in the Values option.

Figure 14. Configuration options for a Fan-out string parameter

A parameter value factory can produce multiple sets of parameter values. If there are multiple sets of parameter values then the downstream job will receive multiple build requests. Where there are multiple parameter value factories producing multiple sets of parameter values, a build request for every combination will be submitted.

By way of example, if there are the following build parameters defined:

A boolean parameter called ACTIVATE value true
A fan-out string parameter called X with values 1, 2 and 3
A string parameter called HOST with value staging
A fan-out string parameter called Y with values A, B and C

Then a total of 9 separate build requests would be made of the downstream job:

ACTIVE = true, X = 1, HOST = staging, Y = A
ACTIVE = true, X = 1, HOST = staging, Y = B
ACTIVE = true, X = 1, HOST = staging, Y = C
ACTIVE = true, X = 2, HOST = staging, Y = A
ACTIVE = true, X = 2, HOST = staging, Y = B
ACTIVE = true, X = 2, HOST = staging, Y = C
ACTIVE = true, X = 3, HOST = staging, Y = A
ACTIVE = true, X = 3, HOST = staging, Y = B
ACTIVE = true, X = 3, HOST = staging, Y = C

Build other remote/local projects post-build action

The Build other remote/local projects post-build action is available for Freestyle jobs.

This post-build action allows you to trigger jobs after a build has completed even in cases where a build step marked the build as a failure and requested that the build stop immediately.

To add the trigger to a Freestyle job:

Navigate to your Freestyle job.
In the left navigation menu, select Configure.
In the left navigation menu, select Post-build Actions.
Select +Add post-build action, and from the dropdown select Build other remote/local jobs.
Select Add downstream job to select the downstream job to trigger.
Configure the downstream job options.
Select Save to save the configuration.

The initial configuration does not include any projects to trigger. The available configuration options are identical to those of the Trigger builds on remote/local jobs build step. You can trigger multiple downstream jobs from the post-build action. All jobs in the post-build action are triggered in parallel.

triggerRemoteJob Pipeline step

The triggerRemoteJob Pipeline step is available for Declarative and Scripted Pipelines.

The triggerRemoteJob Pipeline step allows you to trigger remote jobs from a Pipeline script, and is available when Operations Center Context plugin version 1.8.0 or later is installed.

To add a triggerRemoteJob step to a Pipeline job:

Navigate to your Pipeline job.
In the left navigation menu, select Configure.
In the Pipeline section, select Pipeline Syntax to navigate to the Snippet Generator page.
From the Sample Step dropdown, select triggerRemoteJob: Trigger builds on remote/local jobs.
Configure the parameters for the triggerRemoteJob step.
Select Generate Pipeline Script to generate the Pipeline script snippet.

The triggerRemoteJob step supports the same options as the Trigger builds on remote/local jobs build step, except for the Trigger When parameter, which is not applicable in a Pipeline script.
As with the Trigger builds on remote/local jobs build step, all On… actions have four options, which in Pipeline scripts correspond to:
- ContinueAsIs: Leaves the build result as-is and continues.
- ContinueAsFailure: Sets the build result to FAILURE and continues.
- StopAsFailure: Sets the build result to FAILURE and stops the build without running any further build steps.
- ContinueAsUnstable: Sets the build result to UNSTABLE and continues, unless the build result is already FAILURE.

triggerRemoteJob parameters

The triggerRemoteJob Pipeline step takes the following parameters:

remotePathUrl

Enter the path of the remote job to trigger using one of the following URI formats:

Type / after each path component to fetch the next level of completions.

jenkins://./path/on/this/server: Local paths on the same instance.
cjp:///path/on/operations/center: Paths via the operations center (more legible, but assumes controllers are not moved within operations center).
jenkins://instance-id/path/on/another/controller: Paths on another controller using the instance ID (most stable, but not human-readable).
cjp:///path/to/controller-item/path/on/another/controller: Cross-controller paths via the operations center.

remotePathMissing

The behavior to apply when the remotePathUrl parameter is missing. This can be any of the four options listed above: ContinueAsIs, ContinueAsFailure, StopAsFailure, or ContinueAsUnstable.

mode

Specifies the behavior when waiting on the downstream job and whether the build number of the downstream job is recorded in metadata used by CloudBees Pipeline Explorer.

fireAndForget

Does not wait and assumes the job will be triggered. The build number of the downstream job is not recorded in metadata used by CloudBees Pipeline Explorer.

This is the default value and can be omitted.

Figure 15. fireAndForget

confirmScheduled

Waits for a user-configurable time period, for confirmation of the job being enqueued on the target build queue. The build number of the downstream job is not recorded in metadata used by CloudBees Pipeline Explorer.

Figure 16. confirmScheduled

timeout	how long to wait for the build result of the downstream job.
whenTimeout	select wanted behavior among the ones described in the NOTE section above.

confirmStarted

Waits for a user-configurable time period, for confirmation that the downstream job has started building. The build number of the downstream job is recorded in metadata used by CloudBees Pipeline Explorer; it is available once the downstream job starts, as long as the job starts within the configured timeout time span for the step.

Figure 17. confirmStarted

timeout	how long to wait for the build result of the downstream job.
whenTimeout	select wanted behavior among the ones described in the NOTE section above.

awaitResult

Waits for a user-configurable time period, for confirmation that the target job has completed building. The build number of the downstream job is recorded in metadata used by CloudBees Pipeline Explorer once the downstream build finishes.

Figure 18. awaitResult

timeout	how long to wait for the build result of the downstream job.
whenFailure	what to do when the downstream job fails, select wanted behavior among the ones described in the NOTE section above.
whenTimeout	what to do in case of timeout, select wanted behavior among the ones described in the NOTE section above.
whenUnstable	what to do when the downstream job is unstable, select wanted behavior among the ones described in the NOTE section above.

trackProgressAwaitResult

Allows complete control over monitoring the progression of the build. The build number of the downstream job is recorded in metadata used by CloudBees Pipeline Explorer; it is available once the downstream job starts, as long as the job starts within the configured startedTimeout time span for the step.

Figure 19. trackProgressAwaitResult

scheduledTimeout

How long to wait for confirmation that the build request of the downstream job has been accepted in the build queue.

startedTimeout: How long to wait for confirmation that the downstream job has started building. The default is one hour.
timeout: How long to wait for the build result of the downstream job.
whenFailure: what to do when the downstream fails, select wanted behavior among the ones described in the NOTE section above.
whenScheduledTimeout: what to do when the scheduledTimeout expires, select wanted behavior among the ones described in the NOTE section above.
whenStartedTimeout: what to do when the startedTimeout expires, select wanted behavior among the ones described in the NOTE section above.
whenTimeout: what to do when the timeout expires, select behavior among the ones described in the NOTE section above.
whenUnstable: what to do then the downstream result is unstable, select wanted behavior among the ones described in the NOTE section above.

When the selected mode for the triggerRemoteJob step is either AwaitResult or TrackProgressAwaitResult, the boolean allowAbort option is available and displays as Abort the remote build when aborting the local build in the Pipeline Syntax UI.

Figure 20. AllowAbort option in the Pipeline Syntax UI.

If enabled (true), the downstream job is automatically aborted when the upstream job is aborted.
If disabled (false), the downstream job continues even if the upstream job is aborted.

Figure 21. AllowAbort option in the triggerRemoteJob step.

Trigger jobs from metrics-based alerts

The CloudBees Monitoring plugin provides the ability to monitor various Jenkins metrics and raise alerts when those metrics deviate from user-defined ranges. The default alerter that ships with the CloudBees Monitoring plugin is an email recipient that sends an email containing basic details of the alert.

The Operations Center Context plugin also provides an additional recipient that can be used to trigger a remote project on alert state transitions. The notifier can be used as a global recipient as well as a metric-specific recipient. Some use-cases for this notifier include:

Triggering a project that cleans up the workspaces on a controller when the free disk space falls below a user-defined threshold.
Triggering a cluster operation to schedule a safe restart of a client controller if the free memory has been below 10% for more than one hour.

This functionality requires the following minimum plugin versions:

The Jenkins instance being monitored must be running:
- Operations Center Context plugin 1.8 or later.
- CloudBees Monitoring plugin 2.0 or later.
The Jenkins instance with the target job being triggered must be running Operations Center Context plugin 1.7.0 or later.

If the job being triggered is on a different Jenkins instance from the Jenkins instance that is being monitored, then both Jenkins instances must be part of the same operations center cluster.

To configure a job trigger for a metrics-based alert:

Select in the upper-right corner to navigate to the Manage Jenkins page.
Select System.
Under Alerts, in the Global Recipients section, select + Add.
From the dropdown, select Trigger a build of a remote/local job.

In the Job to trigger field, enter the job path using one of the following URI formats:

Type / after each path component to fetch the next level of completions.

jenkins://./path/on/this/server: Local paths on the same instance.
cjp:///path/on/operations/center: Paths via the operations center (more legible, but assumes controllers are not moved within operations center).
jenkins://instance-id/path/on/another/controller: Paths on another controller using the instance ID (most stable, but not human-readable).

cjp:///path/to/controller-item/path/on/another/controller: Cross-controller paths via the operations center.

By default, the recipient only triggers projects when the alert condition becomes active, and the job request is submitted with a time-to-live of one hour. If the communications path between the source controller and the destination controller is temporarily offline, the build request is retained in the messaging queue for at most one hour.

(Optional) Configure advanced options:

Option Description

Parameter name for source Jenkins URL

If specified, the job is triggered with a parameter set to the URL of the source Jenkins controller from which the alert originates. The default parameter name is SOURCE_JENKINS_URL.

This parameter can be particularly helpful when triggering a job on a remote controller.

Parameter name for condition name

If specified, the job is triggered with a parameter set to the name of the condition that triggered the alert. The default parameter name is CONDITION_NAME.

This parameter can be particularly helpful when a single job receives multiple different alerts.

Parameter name for condition active

If specified, the job is triggered with a parameter set to a boolean value where true indicates the condition has just become active. The default parameter name is CONDITION_ACTIVE.

Trigger the job when condition active

If selected then the job will be triggered when the alert transitions from inactive to active.

Trigger the job when condition inactive

If selected then the job will be triggered when the alert transitions from active to inactive.

Timeout for triggering the job

How long the request for the build request for the job has will remain valid.

If the request cannot be delivered to the downstream controller within this timeout then the build request may be discarded.

If no units of time are specified then the value is assumed to be in seconds. You can specify the timeout in multiple units and the semantics of those units will be preserved (for example, 1h 65s will not be transformed into 3665s). The valid units are:

d, day or days: days
h, hr, hrs, hour or hours: hours
m, min, mins, minute or minutes: minutes
s, sec, secs, second or seconds: seconds
ms, milli, millis, millisecond or milliseconds: milliseconds

The following are all valid timeouts (though they will be simplified on save)

300 will be simplified to 300s i.e. 300 seconds
5 minutes will be simplified to 5m i.e. 300 seconds
4 minutes and 60 seconds will be simplified to 4m60s i.e. 300 seconds
3 minutes and 120 seconds will be simplified to 3m120s i.e. 300 seconds
45secs 3 minutes 75s will be simplified to 3m120s i.e. 300 seconds

Form validation will display the simplified interpretation of the timeout

Select Save to save the configuration.