Depending on the scheme used to organize an operations center cluster, it may be necessary to trigger jobs that are on a remote client controller. For example, the team responsible for the production servers may want to trigger the QA team’s sanity tests against the staging environment servers before deploying to production. Given that the security concerns of the production servers can differ from those of the QA team’s, it may be that the QA team uses a different client controller from the production team. In order to solve these types of problems the Operations Center Context plugin provides a build step and a post build action that can trigger jobs either on the root operations center or any client controller Jenkins instance in the operations center cluster.
|
The ability to trigger jobs across controllers within an operations center cluster has been added to operations center as an update to the 1.6 release. This functionality requires the following minimum plugin versions:
If the operations center server has not been upgraded then
If individual client controllers have not been upgraded then
|
|
The ability to trigger remote jobs is only possible with an adequate level of trust inside the operations center cluster. It is important to understand the Authentication mapping configuration between operations center and the client controllers. |
The Trigger builds on remote projects build step
| The Trigger builds on remote projects build step is available for Freestyle jobs. |
This build steps allows you to intersperse triggering jobs with other build steps. This can be useful if there is a flow of operations needed by the job. For example, you may have a job that resets test database instances. Using this build step you could trigger that job, specify the target database as a build parameter, wait for the reset job to complete and then proceed with the build steps that actually run the tests against the test database.
To add the trigger just select the Trigger builds on remote projects option from the Add build step drop down.
The initial configuration does not include any projects to trigger. You can trigger multiple downstream jobs from the same build step. All jobs in a trigger will be triggered in parallel. If you want to force sequential job triggers you will need to use multiple build steps.
Clicking the Add button will allow you to select the job to trigger and configure the options on the trigger. The Add button can be used repeatedly if you have multiple jobs to trigger.
|
All jobs in the same Trigger builds on remote projects build step are processed in parallel. If you have a build step
with two jobs where the first job is configured to wait until finished and mark the build as a failure if the downstream
job failed and the second job is configured to trigger only when successful, both jobs will be triggered even if the
first downstream job fails as the triggering occurs before the results of execution are known. However if the first
job is deleted then the On job missing configuration may mark the build as |
Configuring the downstream job
There are four things that must be specified for each downstream job:
-
Job Name - this uses a special path browser UI control to allow you to select any job visible to you throughout the operations center cluster.
Figure 3. Selecting a job to trigger
Figure 4. Job to trigger selectedThe path browser UI provides visual indicators of its state.
Until an item has been selected, the control will be blue with no confirmation icon in the right hand side of the input box.
Figure 5. A job selection control where a selection has not been completedYou can either navigate with the keyboard or mouse. Typing the name of the item can be used to select it or refine the list of items displayed. Additionally the entire
/separated path can be pasted into the control.Once a valid selection has been made the control should turn green and a check box icon should appear in the right hand side of the input box.
Figure 6. A job selection control where a selection has been completedWhen viewing/modifying the configuration of an existing job there may be occasions when either the communications path to the remote Jenkins is off-line or where the user viewing/modifying does not have permission to view the already selected downstream job. In such cases the control will display a partial path and a warning which provides a
Retrylink.In the event that the warning was shown because of an interrupted communications channel, the retry link will restore the controls functionality.
As it is not possible to reveal whether the downstream job does not exist or whether the current user just does not have permission to view the downstream job the control will remain in a read-only state while displaying a warning.
Figure 7. A job selection control where some of the nodes to the selected item are currently off-line -
Trigger When - the allows you to choose, based on the current build state, when the trigger should run:
Figure 8. The available Trigger When options-
Always will trigger the selected job irrespective of the current build result
-
Either unstable or successful will trigger the selected job if the current build has been marked as
UNSTABLEor retains the initial build result ofSUCCESSbut will not trigger the selected job if a previous build step has marked the build asFAILED. -
Only when successful will trigger the selected job only if the current build retains the the initial build result of
SUCCESS. If a previous build step marked the build as eitherUNSTABLEorFAILEDthen the job will not be triggered.
-
-
On job missing - this allows you to choose what should happen to the current build if the specified job cannot be located (i.e. if it has been moved / deleted)
-
Mode - this allows specifying any blocking behavior with regards to the triggered job.
Figure 9. The available Mode options-
Fire and forget does not wait and assumes that the job will be triggered. The build request will be sent with a 24h time-to-live, so as long as the Jenkins instance sending the trigger connects to the operations center server (in order for the request to be forwarded to the operations center server messaging transport) and the target Jenkins instance subsequently connects to the operations center server within that 24h window the job will be triggered.
Figure 10. Fire and forget mode options -
Wait until scheduled will wait, for a user configurable time period, for confirmation of the job being enqueued on the target Jenkins build queue. The build request will be sent with a time-to-live of the specified timeout. That is, if you specify a 5 minute timeout the build request will expire after 5 minutes, if you specify a 20 day timeout then the build request will expire after 20 days, etc.
If the confirmation is not received within the timeout, then the On timeout action will be applied to the build.
Multiple requests to build the downstream job using this mode can be coalesced by the downstream Jenkins.
Figure 11. Wait until scheduled mode options -
Wait until started will wait, for a user configurable time period, for confirmation that the target job has started building. The build request will be sent with a time-to-live of the specified timeout. That is, if you specify a 5 minute timeout the build request will expire after 5 minutes, if you specify a 20 day timeout then the build request will expire after 20 days, etc.
If the confirmation is not received within the timeout, then the On timeout action will be applied to the build.
Once confirmation is received the downstream build number will be recorded on the build log.
Multiple requests to build the downstream job using this mode can be coalesced by the downstream Jenkins.
Figure 12. Wait until started mode options -
Wait until finished will wait, for a user configurable time period, for confirmation that the target job has completed building. The build request will be sent with a time-to-live of 24h.
If the build result is not received within the timeout, then the On timeout action will be applied to the build.
When the build result is received, the downstream build number and the build result will be recorded on the build log.
If the build result is
UNSTABLEthen the On unstable action will be applied to the build.If the build result is
FAILEDthen the On failure action will be applied to the build.Multiple requests to build the downstream job using this mode are not coalesced by the downstream Jenkins as the requirement to report back the tracking information ensures separate builds for each request.
Figure 13. Wait until finished mode options -
Track progress and wait until finished allows complete control over monitoring the progression of the build. The Scheduled timeout controls how long to wait for the confirmation that the build is enqueued. This timeout also controls the time-to-live of the build request. If left blank then a 24h time-to-live will be used. In the event of the scheduled timeout expiring, the On scheduled timeout action will be applied to the build.
The Started timeout controls how long to wait for confirmation of the downstream job starting. If confirmation is not received within the specified timeout then the On started timeout action will be applied to the build. If confirmation is received then the downstream build number will be recorded on the build log.
Finally the Finished timeout controls how long to wait for confirmation that the target job has completed building.
If the build result is not received within the timeout, then the On finished timeout action will be applied to the build.
When the build result is received, the downstream build number and the build result will be recorded on the build log.
If the build result is
UNSTABLEthen the On unstable action will be applied to the build.If the build result is
FAILEDthen the On failure action will be applied to the build.Multiple requests to build the downstream job using this mode are not coalesced by the downstream Jenkins as the requirement to report back the tracking information ensures separate builds for each request.
The Track progress and wait until finished mode provides real-time state details about the downstream job or jobs being triggered, including a clickable job path and build number that provide a convenient way to see job details. Refer to the example below.
To include this output in your build logs, you must specify a timeout value for each of the job status states you want to track.
starting remote job [Pipeline] triggerRemoteJob [operations-center-context] Triggering Controller1 » localTarget [operations-center-context] Waiting for triggered job [operations-center-context] Controller1 » localTarget [operations-center-context] to be SCHEDULED... [operations-center-context] Controller1 » localTarget SCHEDULED [operations-center-context] Finished waiting for triggered job(s) to be SCHEDULED after 0 ms [operations-center-context] Waiting for triggered job [operations-center-context] Controller1 » localTarget [operations-center-context] to be STARTED... [operations-center-context] Controller1 » localTarget #16 STARTED [operations-center-context] Finished waiting for triggered job(s) to be STARTED after 9 sec [operations-center-context] Waiting for triggered job [operations-center-context] Controller1 » localTarget [operations-center-context] to be FINISHED... [operations-center-context] Controller1 » localTarget #16 SUCCESS [operations-center-context] Finished waiting for triggered job(s) to be FINISHED after 1.7 sec
Figure 14. Track progress and wait until finished mode optionsEach of the On … actions has four options:
-
Ignore and continue - which will leave the build result as is and continue.
-
Mark build as failure and continue - which will set the build result to
FAILUREand continue. -
Mark build as failure and stop - which will set the build result to
FAILUREand try to stop the build without running any further build steps. -
Mark build as unstable and continue - which will set the build result to
UNSTABLE- unless it is already marked asFAILURE- and continue.
Figure 15. Configuring failure modes -
-
Triggering parameterized jobs
In addition to the four mandatory configuration options for each downstream job you also have the option of specifying the build parameters that the job will be supplied with when building.
|
Parameters on a trigger for a non-parameterized downstream job will be stripped from the build request on receipt by the downstream Jenkins. Similarly, any parameters that are not defined in a parameterized downstream job will be stripped from the build request. This is to ensure that a trigger cannot maliciously manipulate the downstream build job’s environment as most build parameters get exposed as environment variables in the build run. |
The Add parameters button can be used to add parameter value factories to the build request(s) for the downstream job. The parameter value factories are an extension point.
The Operations Center Context plugin provides the following parameter value factories:
-
Boolean parameter produces a single parameter value which can be either true or false;
Figure 18. The configuration options for a Boolean parameter -
Current build parameters produces a single set of parameters that are a subset of the triggering job’s build parameters. This allows you to propagate the triggering job’s build parameters to the triggered job.
By default parameters which are defined as having sensitive values - such as password parameters - will be excluded but the Parameters with sensitive values option allows this to be configured.
If you need to exclude some of the build parameters from the trigger use the Excluded parameters option. This takes a multi-line list of parameter names (one per line) which will be excluded from the build request. Wildcards matching is supported by using the
*character.
Figure 19. The configuration options for Current build parameters -
String parameter produces a single parameter value specified as a verbatim string constant.
Figure 20. The configuration options for a String parameter -
Fan-out string parameter produces multiple sets of parameter values. This will result in multiple build requests of the downstream job. Each value is specified on a separate line in the Values option.
Figure 21. The configuration options for a Fan-out string parameter
|
A parameter value factory can produce multiple sets of parameter values. If there are multiple sets of parameter values then the downstream job will receive multiple build requests. Where there are multiple parameter value factories producing multiple sets of parameter values, a build request for every combination will be submitted. By way of example, if there are the following build parameters defined:
Then a total of 9 separate build requests would be made of the downstream job:
|
The Build other remote/local projects post-build action
| The Build other remote/local projects post-build action is available for Freestyle jobs. |
This post-build action allows you to trigger jobs after a build has completed even in those cases where a build step marked the build as a failure and requested that the build stop immediately.
To add the trigger just select the Builds other remote projects option from the Add post-build action drop down.
The initial configuration does not include any projects to trigger. The available configuration options are identical to those of the Trigger builds on remote/local projects build step. You can trigger multiple downstream jobs from the post-build action. All jobs in the post-build action will be triggered in parallel.
The Trigger Remote Job Pipeline Step
|
The Trigger Pipeline step is available for declarative and scripted pipelines. |
As of operations-center-context version 1.8.0, it is possible to trigger remote jobs from a Pipeline job using the ad-hoc Remote Trigger Job function. The RemoteTriggerJob step will appear in the list of available steps of Pipeline if operations-center-context (v. 1.8.0 +) plugin is installed in the Jenkins Instance.
All the parameters available in the remote trigger functionality explained above will be available in the Pipeline step, except for the when parameter which is not applicable for a Pipeline script.
|
Just as the configuration described in the previous section, all the On… actions has four options, which in Pipeline scripts correspond to:
|
The remote trigger Pipeline step takes the following parameters:
- remotePathUrl
-
The URL of the remote Job to trigger. This can have 2 formats
-
jenkins://jenkins_instance_id/path_to_job: if the job to trigger is in the same instance where the workflow job is configured, it is possible to use.to indicate the current instance. -
cjp://path_from_root_of_cjoc: the walking path to the job from the root of operations center.
-
- remotePathMissing
-
The behavior to take when the remotePathUrl parameter is missing. This can be any of the behavior described in the NOTE section above.
- mode
-
The kind of blocking behavior wanted with respect to waiting on the downstream job
FireAndForget does not wait and assumes that the job will be triggered. This is the default value and can be omitted.
ConfirmScheduled will wait, for a user configurable time period, for confirmation of the job being enqueued on the target Jenkins build queue.
timeout how long to wait for the build result of the downstream job.
whenTimeout select wanted behavior among the ones described in the NOTE section above.
ConfirmStarted will wait, for a user configurable time period, for confirmation that the target job has started building.
timeout how long to wait for the build result of the downstream job.
whenTimeout select wanted behavior among the ones described in the NOTE section above.
AwaitResult will wait, for a user configurable time period, for confirmation that the target job has completed building.
timeout how long to wait for the build result of the downstream job.
whenFailure what to do when the downstream job fails, select wanted behavior among the ones described in the NOTE section above.
whenTimeout what to do in case of timeout, select wanted behavior among the ones described in the NOTE section above.
whenUnstable what to do when the downstream job is unstable, select wanted behavior among the ones described in the NOTE section above.
TrackProgressAwaitResult allows complete control over monitoring the progression of the build.
scheduledTimeout How long to wait for confirmation that the build request of the downstream job has been accepted in the build queue.
startedTimeout How long to wait for confirmation that the downstream job has started building.
timeout How long to wait for the build result of the downstream job.
whenFailure what to do when the downstream fails, select wanted behavior among the ones described in the NOTE section above.
whenScheduledTimeout what to do when the scheduledTimeout expires, select wanted behavior among the ones described in the NOTE section above.
whenStartedTimeout what to do when the startedTimeout expires, select wanted behavior among the ones described in the NOTE section above.
whenTimeout what to do when the timeout expires, select behavior among the ones described in the NOTE section above.
whenUnstable what to do then the downstream result is unstable, select wanted behavior among the ones described in the NOTE section above.
Triggering jobs from metrics based alerts
The CloudBees Monitoring plugin provides the ability to monitor various metrics of Jenkins and raise alerts when those metrics deviate from user defined ranges. The default alerter that ships with the CloudBees Monitoring plugin is an email recipient that will send an email containing basic details of the alert. The Operations Center Context plugin also provides an additional recipient that can be used to trigger a remote project on alert state transitions.
|
This functionality requires the following minimum plugin versions:
If the job being triggered is on a different Jenkins instance from the Jenkins instances that is being monitored, then both Jenkins instances must be part of the same operations center cluster. |
The notifier can be used as a global recipient as well as a metric specific recipient.
Some use-cases for this notifier could include:
-
Triggering a project that cleans up the workspaces on a controller when the free disk space falls below a user defined threshold
-
Triggering a cluster operation to schedule a safe restart of a client controller if the free memory has been below 10% for more than 1 hour
-
etc
By default:
-
The recipient will only trigger projects when the alert condition becomes active.
-
If the project being triggered is parameterized the following parameter values will be provided by default:
If the project being triggered does not have a parameter with the corresponding name then the parameter value will be ignored when the job is triggered. -
SOURCE_JENKINS_URLwhich will contain a string value corresponding to the URL of the root of the Jenkins controller from which the alert originates. -
CONDITION_NAMEwhich will contain a string value corresponding to the name of the condition that has triggered. -
CONDITION_ACTIVEwhich will contain a boolean value where truth indicates that the condition has just transitioned to active.
-
-
The job request will be submitted with a time-to-live of 1 hour. i.e. if the communications path between the source controller and the destination controller is temporarily off-line, the build request will be retained in the messaging queue for at most 1 hour.
The default options can be customized using the advanced options.
Option |
Description |
||
|---|---|---|---|
Job to trigger |
The job to trigger |
||
Parameter name for source Jenkins URL |
If specified then job will be triggered with this named parameter having the value of this Jenkins controller’s URL.
|
||
Parameter name for condition name |
If specified then job will be triggered with this named parameter having this alert’s title.
|
||
Parameter name for condition active |
If specified then job will be triggered with this named parameter having indicating whether the alert is currently active or inactive. |
||
Trigger the job when condition active |
If selected then the job will be triggered when the alert transitions from inactive to active. |
||
Trigger the job when condition inactive |
If selected then the job will be triggered when the alert transitions from active to inactive. |
||
Timeout for triggering the job |
How long the request for the build request for the job has will remain valid. If the request cannot be delivered to the downstream controller within this timeout then the build request may be discarded. If no units of time are specified then the value is assumed to be in seconds.
You can specify the timeout in multiple units and the symantics of those units will be preserved, e.g.
The following are all valid timeouts (though they will be simplified on save)
|