Cluster-wide job triggers

16 minute read

Depending on the scheme used to organize an operations center cluster, it may be necessary to trigger jobs that are on a remote client controller. For example, the team responsible for the production servers may want to trigger the QA team’s sanity tests against the staging environment servers before deploying to production. Given that the security concerns of the production servers can differ from those of the QA team’s, it may be that the QA team uses a different client controller from the production team. In order to solve these types of problems the Operations Center Context plugin provides a build step and a post build action that can trigger jobs either on the root operations center or any client controller Jenkins instance in the operations center cluster.

The ability to trigger jobs across controllers within an operations center cluster has been added to operations center as an update to the 1.6 release. This functionality requires the following minimum plugin versions:

  • Client controllers must be running

    • operations-center-context 1.6.1 or newer

    • operations-center-client 1.6.1 or newer

  • Operations center server must be running

    • operations-center-context 1.6.1 or newer

    • operations-center-server 1.6.4 or newer

    • operations-center-sso 1.6.1 or newer

If the operations center server has not been upgraded then

  • client controllers will be unable to trigger jobs on remote controllers

If individual client controllers have not been upgraded then

  • those client controllers will be unable to trigger jobs on remote controllers; and

  • other remote controllers will be able to trigger jobs on the client controllers with operations-center-context 1.6 but the blocking behavior and parameterized trigger functionality will be silently unavailable for jobs running on the client controllers with operations-center-context 1.6.

The ability to trigger remote jobs is only possible with an adequate level of trust inside the operations center cluster. It is important to understand the Authentication mapping configuration between operations center and the client controllers.

The Trigger builds on remote projects build step

The Trigger builds on remote projects build step is available for Freestyle jobs.

This build steps allows you to intersperse triggering jobs with other build steps. This can be useful if there is a flow of operations needed by the job. For example, you may have a job that resets test database instances. Using this build step you could trigger that job, specify the target database as a build parameter, wait for the reset job to complete and then proceed with the build steps that actually run the tests against the test database.

To add the trigger just select the Trigger builds on remote projects option from the Add build step drop down.

Figure 1. Adding a build step to trigger jobs across the operations center cluster
Figure 1. Adding a build step to trigger jobs across the operations center cluster

The initial configuration does not include any projects to trigger. You can trigger multiple downstream jobs from the same build step. All jobs in a trigger will be triggered in parallel. If you want to force sequential job triggers you will need to use multiple build steps.

Figure 2. An empty Trigger builds on remote projects build step
Figure 2. An empty Trigger builds on remote projects build step

Clicking the Add button will allow you to select the job to trigger and configure the options on the trigger. The Add button can be used repeatedly if you have multiple jobs to trigger.

All jobs in the same Trigger builds on remote projects build step are processed in parallel. If you have a build step with two jobs where the first job is configured to wait until finished and mark the build as a failure if the downstream job failed and the second job is configured to trigger only when successful, both jobs will be triggered even if the first downstream job fails as the triggering occurs before the results of execution are known. However if the first job is deleted then the On job missing configuration may mark the build as FAILED and prevent the second job from being triggered.

Configuring the downstream job

There are four things that must be specified for each downstream job:

  • Job Name - this uses a special path browser UI control to allow you to select any job visible to you throughout the operations center cluster.

    builder with job2
    Figure 3. Selecting a job to trigger
    builder with job3
    Figure 4. Job to trigger selected

    The path browser UI provides visual indicators of its state.

    Until an item has been selected, the control will be blue with no confirmation icon in the right hand side of the input box.

    Figure 5. A job selection control where a selection has not been completed
    Figure 5. A job selection control where a selection has not been completed

    You can either navigate with the keyboard or mouse. Typing the name of the item can be used to select it or refine the list of items displayed. Additionally the entire / separated path can be pasted into the control.

    Once a valid selection has been made the control should turn green and a check box icon should appear in the right hand side of the input box.

    Figure 6. A job selection control where a selection has been completed
    Figure 6. A job selection control where a selection has been completed

    When viewing/modifying the configuration of an existing job there may be occasions when either the communications path to the remote Jenkins is off-line or where the user viewing/modifying does not have permission to view the already selected downstream job. In such cases the control will display a partial path and a warning which provides a Retry link.

    In the event that the warning was shown because of an interrupted communications channel, the retry link will restore the controls functionality.

    As it is not possible to reveal whether the downstream job does not exist or whether the current user just does not have permission to view the downstream job the control will remain in a read-only state while displaying a warning.

    path segments offline
    Figure 7. A job selection control where some of the nodes to the selected item are currently off-line
  • Trigger When - the allows you to choose, based on the current build state, when the trigger should run:

    configure trigger when
    Figure 8. The available Trigger When options
    • Always will trigger the selected job irrespective of the current build result

    • Either unstable or successful will trigger the selected job if the current build has been marked as UNSTABLE or retains the initial build result of SUCCESS but will not trigger the selected job if a previous build step has marked the build as FAILED.

    • Only when successful will trigger the selected job only if the current build retains the the initial build result of SUCCESS. If a previous build step marked the build as either UNSTABLE or FAILED then the job will not be triggered.

  • On job missing - this allows you to choose what should happen to the current build if the specified job cannot be located (i.e. if it has been moved / deleted)

  • Mode - this allows specifying any blocking behavior with regards to the triggered job.

    configure blocking mode
    Figure 9. The available Mode options
    • Fire and forget does not wait and assumes that the job will be triggered. The build request will be sent with a 24h time-to-live, so as long as the Jenkins instance sending the trigger connects to the operations center server (in order for the request to be forwarded to the operations center server messaging transport) and the target Jenkins instance subsequently connects to the operations center server within that 24h window the job will be triggered.

      blocking mode fire and forget
      Figure 10. Fire and forget mode options
    • Wait until scheduled will wait, for a user configurable time period, for confirmation of the job being enqueued on the target Jenkins build queue. The build request will be sent with a time-to-live of the specified timeout. That is, if you specify a 5 minute timeout the build request will expire after 5 minutes, if you specify a 20 day timeout then the build request will expire after 20 days, etc.

      If the confirmation is not received within the timeout, then the On timeout action will be applied to the build.

      Multiple requests to build the downstream job using this mode can be coalesced by the downstream Jenkins.

      blocking mode wait scheduled
      Figure 11. Wait until scheduled mode options
    • Wait until started will wait, for a user configurable time period, for confirmation that the target job has started building. The build request will be sent with a time-to-live of the specified timeout. That is, if you specify a 5 minute timeout the build request will expire after 5 minutes, if you specify a 20 day timeout then the build request will expire after 20 days, etc.

      If the confirmation is not received within the timeout, then the On timeout action will be applied to the build.

      Once confirmation is received the downstream build number will be recorded on the build log.

      Multiple requests to build the downstream job using this mode can be coalesced by the downstream Jenkins.

      blocking mode wait started
      Figure 12. Wait until started mode options
    • Wait until finished will wait, for a user configurable time period, for confirmation that the target job has completed building. The build request will be sent with a time-to-live of 24h.

      If the build result is not received within the timeout, then the On timeout action will be applied to the build.

      When the build result is received, the downstream build number and the build result will be recorded on the build log.

      If the build result is UNSTABLE then the On unstable action will be applied to the build.

      If the build result is FAILED then the On failure action will be applied to the build.

      Multiple requests to build the downstream job using this mode are not coalesced by the downstream Jenkins as the requirement to report back the tracking information ensures separate builds for each request.

      blocking mode wait finished
      Figure 13. Wait until finished mode options
    • Track progress and wait until finished allows complete control over monitoring the progression of the build. The Scheduled timeout controls how long to wait for the confirmation that the build is enqueued. This timeout also controls the time-to-live of the build request. If left blank then a 24h time-to-live will be used. In the event of the scheduled timeout expiring, the On scheduled timeout action will be applied to the build.

      The Started timeout controls how long to wait for confirmation of the downstream job starting. If confirmation is not received within the specified timeout then the On started timeout action will be applied to the build. If confirmation is received then the downstream build number will be recorded on the build log.

      Finally the Finished timeout controls how long to wait for confirmation that the target job has completed building.

      If the build result is not received within the timeout, then the On finished timeout action will be applied to the build.

      When the build result is received, the downstream build number and the build result will be recorded on the build log.

      If the build result is UNSTABLE then the On unstable action will be applied to the build.

      If the build result is FAILED then the On failure action will be applied to the build.

      Multiple requests to build the downstream job using this mode are not coalesced by the downstream Jenkins as the requirement to report back the tracking information ensures separate builds for each request.

      The Track progress and wait until finished mode provides real-time state details about the downstream job or jobs being triggered, including a clickable job path and build number that provide a convenient way to see job details. Refer to the example below.

      To include this output in your build logs, you must specify a timeout value for each of the job status states you want to track.

      starting remote job [Pipeline] triggerRemoteJob [operations-center-context] Triggering Controller1 » localTarget [operations-center-context] Waiting for triggered job [operations-center-context] Controller1 » localTarget [operations-center-context] to be SCHEDULED... [operations-center-context] Controller1 » localTarget SCHEDULED [operations-center-context] Finished waiting for triggered job(s) to be SCHEDULED after 0 ms [operations-center-context] Waiting for triggered job [operations-center-context] Controller1 » localTarget [operations-center-context] to be STARTED... [operations-center-context] Controller1 » localTarget #16 STARTED [operations-center-context] Finished waiting for triggered job(s) to be STARTED after 9 sec [operations-center-context] Waiting for triggered job [operations-center-context] Controller1 » localTarget [operations-center-context] to be FINISHED... [operations-center-context] Controller1 » localTarget #16 SUCCESS [operations-center-context] Finished waiting for triggered job(s) to be FINISHED after 1.7 sec
      blocking mode track progress
      Figure 14. Track progress and wait until finished mode options

      Each of the On …​ actions has four options:

      • Ignore and continue - which will leave the build result as is and continue.

      • Mark build as failure and continue - which will set the build result to FAILURE and continue.

      • Mark build as failure and stop - which will set the build result to FAILURE and try to stop the build without running any further build steps.

      • Mark build as unstable and continue - which will set the build result to UNSTABLE - unless it is already marked as FAILURE - and continue.

      configure failure mode
      Figure 15. Configuring failure modes

Triggering parameterized jobs

In addition to the four mandatory configuration options for each downstream job you also have the option of specifying the build parameters that the job will be supplied with when building.

Parameters on a trigger for a non-parameterized downstream job will be stripped from the build request on receipt by the downstream Jenkins.

Similarly, any parameters that are not defined in a parameterized downstream job will be stripped from the build request.

This is to ensure that a trigger cannot maliciously manipulate the downstream build job’s environment as most build parameters get exposed as environment variables in the build run.

The Add parameters button can be used to add parameter value factories to the build request(s) for the downstream job. The parameter value factories are an extension point.

job parameters 1
Figure 16. Configuring downstream job parameters
job parameters 2
Figure 17. A configured downstream job parameter

The Operations Center Context plugin provides the following parameter value factories:

  • Boolean parameter produces a single parameter value which can be either true or false;

    job param boolean
    Figure 18. The configuration options for a Boolean parameter
  • Current build parameters produces a single set of parameters that are a subset of the triggering job’s build parameters. This allows you to propagate the triggering job’s build parameters to the triggered job.

    By default parameters which are defined as having sensitive values - such as password parameters - will be excluded but the Parameters with sensitive values option allows this to be configured.

    If you need to exclude some of the build parameters from the trigger use the Excluded parameters option. This takes a multi-line list of parameter names (one per line) which will be excluded from the build request. Wildcards matching is supported by using the * character.

    job param current
    Figure 19. The configuration options for Current build parameters
  • String parameter produces a single parameter value specified as a verbatim string constant.

    job param string
    Figure 20. The configuration options for a String parameter
  • Fan-out string parameter produces multiple sets of parameter values. This will result in multiple build requests of the downstream job. Each value is specified on a separate line in the Values option.

    job param fan out
    Figure 21. The configuration options for a Fan-out string parameter

A parameter value factory can produce multiple sets of parameter values. If there are multiple sets of parameter values then the downstream job will receive multiple build requests. Where there are multiple parameter value factories producing multiple sets of parameter values, a build request for every combination will be submitted.

By way of example, if there are the following build parameters defined:

  • A boolean parameter called ACTIVATE value true

  • A fan-out string parameter called X with values 1, 2 and 3

  • A string parameter called HOST with value staging

  • A fan-out string parameter called Y with values A, B and C

Then a total of 9 separate build requests would be made of the downstream job:

  • ACTIVE = true, X = 1, HOST = staging, Y = A

  • ACTIVE = true, X = 1, HOST = staging, Y = B

  • ACTIVE = true, X = 1, HOST = staging, Y = C

  • ACTIVE = true, X = 2, HOST = staging, Y = A

  • ACTIVE = true, X = 2, HOST = staging, Y = B

  • ACTIVE = true, X = 2, HOST = staging, Y = C

  • ACTIVE = true, X = 3, HOST = staging, Y = A

  • ACTIVE = true, X = 3, HOST = staging, Y = B

  • ACTIVE = true, X = 3, HOST = staging, Y = C

The Build other remote/local projects post-build action

The Build other remote/local projects post-build action is available for Freestyle jobs.

This post-build action allows you to trigger jobs after a build has completed even in those cases where a build step marked the build as a failure and requested that the build stop immediately.

To add the trigger just select the Builds other remote projects option from the Add post-build action drop down.

Figure 22. Adding a post-build action to trigger jobs across the operations center cluster at the end of the build
Figure 22. Adding a post-build action to trigger jobs across the operations center cluster at the end of the build

The initial configuration does not include any projects to trigger. The available configuration options are identical to those of the Trigger builds on remote/local projects build step. You can trigger multiple downstream jobs from the post-build action. All jobs in the post-build action will be triggered in parallel.

Figure 23. A Build other remote/local projects post-build action added
Figure 23. A Build other remote/local projects post-build action added

The Trigger Remote Job Pipeline Step

The Trigger Pipeline step is available for declarative and scripted pipelines.

As of operations-center-context version 1.8.0, it is possible to trigger remote jobs from a Pipeline job using the ad-hoc Remote Trigger Job function. The RemoteTriggerJob step will appear in the list of available steps of Pipeline if operations-center-context (v. 1.8.0 +) plugin is installed in the Jenkins Instance.

All the parameters available in the remote trigger functionality explained above will be available in the Pipeline step, except for the when parameter which is not applicable for a Pipeline script.

remote trigger job workflow step parameters
Figure 24. Trigger remote job sample step

Just as the configuration described in the previous section, all the On…​ actions has four options, which in Pipeline scripts correspond to:

  • ContinueAsIs: which will leave the build result as is and continue.

  • ContinueAsFailure: which will set the build result to FAILURE and continue.

  • StopAsFailure: which will set the build result to FAILURE and try to stop the build without running any further build steps.

  • ContinueAsUnstable: which will set the build result to UNSTABLE - unless it is already marked as FAILURE - and continue.

The remote trigger Pipeline step takes the following parameters:

remotePathUrl

The URL of the remote Job to trigger. This can have 2 formats

  • jenkins://jenkins_instance_id/path_to_job: if the job to trigger is in the same instance where the workflow job is configured, it is possible to use . to indicate the current instance.

  • cjp://path_from_root_of_cjoc: the walking path to the job from the root of operations center.

remotePathMissing

The behavior to take when the remotePathUrl parameter is missing. This can be any of the behavior described in the NOTE section above.

mode

The kind of blocking behavior wanted with respect to waiting on the downstream job

FireAndForget

does not wait and assumes that the job will be triggered. This is the default value and can be omitted.

remote trigger job fire and forget
ConfirmScheduled

will wait, for a user configurable time period, for confirmation of the job being enqueued on the target Jenkins build queue.

remote trigger job confirm scheduled
timeout

how long to wait for the build result of the downstream job.

whenTimeout

select wanted behavior among the ones described in the NOTE section above.

ConfirmStarted

will wait, for a user configurable time period, for confirmation that the target job has started building.

remote trigger job confirm started
timeout

how long to wait for the build result of the downstream job.

whenTimeout

select wanted behavior among the ones described in the NOTE section above.

AwaitResult

will wait, for a user configurable time period, for confirmation that the target job has completed building.

remote trigger job await result
timeout

how long to wait for the build result of the downstream job.

whenFailure

what to do when the downstream job fails, select wanted behavior among the ones described in the NOTE section above.

whenTimeout

what to do in case of timeout, select wanted behavior among the ones described in the NOTE section above.

whenUnstable

what to do when the downstream job is unstable, select wanted behavior among the ones described in the NOTE section above.

TrackProgressAwaitResult

allows complete control over monitoring the progression of the build.

remote trigger job track result
scheduledTimeout

How long to wait for confirmation that the build request of the downstream job has been accepted in the build queue.

startedTimeout

How long to wait for confirmation that the downstream job has started building.

timeout

How long to wait for the build result of the downstream job.

whenFailure

what to do when the downstream fails, select wanted behavior among the ones described in the NOTE section above.

whenScheduledTimeout

what to do when the scheduledTimeout expires, select wanted behavior among the ones described in the NOTE section above.

whenStartedTimeout

what to do when the startedTimeout expires, select wanted behavior among the ones described in the NOTE section above.

whenTimeout

what to do when the timeout expires, select behavior among the ones described in the NOTE section above.

whenUnstable

what to do then the downstream result is unstable, select wanted behavior among the ones described in the NOTE section above.

Triggering jobs from metrics based alerts

The CloudBees Monitoring plugin provides the ability to monitor various metrics of Jenkins and raise alerts when those metrics deviate from user defined ranges. The default alerter that ships with the CloudBees Monitoring plugin is an email recipient that will send an email containing basic details of the alert. The Operations Center Context plugin also provides an additional recipient that can be used to trigger a remote project on alert state transitions.

This functionality requires the following minimum plugin versions:

  • Jenkins instance being monitored must be

    • operations-center-context 1.8 or newer

    • cloudbees-monitoring 2.0 or newer

  • Jenkins instance with the target job being triggered must be running

    • operations-center-context 1.7.0 or newer

If the job being triggered is on a different Jenkins instance from the Jenkins instances that is being monitored, then both Jenkins instances must be part of the same operations center cluster.

The notifier can be used as a global recipient as well as a metric specific recipient.

Some use-cases for this notifier could include:

  • Triggering a project that cleans up the workspaces on a controller when the free disk space falls below a user defined threshold

  • Triggering a cluster operation to schedule a safe restart of a client controller if the free memory has been below 10% for more than 1 hour

  • etc

Figure 25. Adding a Trigger a build of a remote/local project global recipient to alerting
Figure 25. Adding a Trigger a build of a remote/local project global recipient to alerting

By default:

  • The recipient will only trigger projects when the alert condition becomes active.

  • If the project being triggered is parameterized the following parameter values will be provided by default:

    If the project being triggered does not have a parameter with the corresponding name then the parameter value will be ignored when the job is triggered.
    • SOURCE_JENKINS_URL which will contain a string value corresponding to the URL of the root of the Jenkins controller from which the alert originates.

    • CONDITION_NAME which will contain a string value corresponding to the name of the condition that has triggered.

    • CONDITION_ACTIVE which will contain a boolean value where truth indicates that the condition has just transitioned to active.

  • The job request will be submitted with a time-to-live of 1 hour. i.e. if the communications path between the source controller and the destination controller is temporarily off-line, the build request will be retained in the messaging queue for at most 1 hour.

The default options can be customized using the advanced options.

Figure 26. The advanced options for Trigger a build of a remote/local project
Figure 26. The advanced options for Trigger a build of a remote/local project
Table 1. The configuration options available for Trigger a build of a remote/local project

Option

Description

Job to trigger

The job to trigger

Parameter name for source Jenkins URL

If specified then job will be triggered with this named parameter having the value of this Jenkins controller’s URL.

This parameter can be particularly helpful when triggering a job on a remote controller.

Parameter name for condition name

If specified then job will be triggered with this named parameter having this alert’s title.

This parameter can be particularly helpful when a single job receives multiple different alerts.

Parameter name for condition active

If specified then job will be triggered with this named parameter having indicating whether the alert is currently active or inactive.

Trigger the job when condition active

If selected then the job will be triggered when the alert transitions from inactive to active.

Trigger the job when condition inactive

If selected then the job will be triggered when the alert transitions from active to inactive.

Timeout for triggering the job

How long the request for the build request for the job has will remain valid.

If the request cannot be delivered to the downstream controller within this timeout then the build request may be discarded.

If no units of time are specified then the value is assumed to be in seconds. You can specify the timeout in multiple units and the symantics of those units will be preserved, e.g. 1h 65s will not be transformed into 3665s. The valid units are:

d, day or days

days

h, hr, hrs, hour or hours

hours

m, min, mins, minute or minutes

minutes

s, sec, secs, second or seconds

seconds

ms, milli, millis, millisecond or milliseconds

milliseconds

The following are all valid timeouts (though they will be simplified on save)

  • 300 will be simplified to 300s i.e. 300 seconds

  • 5 minutes will be simplified to 5m i.e. 300 seconds

  • 4 minutes and 60 seconds will be simplified to 4m60s i.e. 300 seconds

  • 3 minutes and 120 seconds will be simplified to 3m120s i.e. 300 seconds

  • 45secs 3 minutes 75s will be simplified to 3m120s i.e. 300 seconds

Form validation will display the simplified interpretation of the timeout