Question
Preconditions/wait-until implementations - how do they get processed in a jobStep or Pipeline?
Answer
A Pipeline waitUntil behaviour is similar to using a procedure step precondition, so the remainder of this article will use the term "precondition".
As you might expect, a precondition construct is setup to allow an active job/pipeline to be put into a holding pattern until the defined logic in the precondition definition evaluates to TRUE.
Be aware that a precondition is separate from a "runCondition" or "run If", whose logic check confirms whether the associated step/task should be executed or skipped.
The logic you place into this definition is typically going to be either a single property (eg: $[/some/path/to/a/property]) or a $[/javascript ....] expression that uses more complex logic, perhaps relying on multiple properties.
When a step is ready to be activated, the precondition code is evaluated prior to the step status being marked as "runnable". Success will allow the step/process to move forward to look for a resource assignment by the step scheduler. If instead the logic check evaluates to FALSE, then this precondition will be placed onto a "review list" that identifies any properties used inside the logic expression. Whenever any of these properties encounter a change, then the associated precondition logic will be re-evaluated. If the logic then evaluates to TRUE, the step will become "runnable", and this logic check will be taken off of the "review list".
| An important fact about preconditions - you must avoid writing logic checks to be solely based on values that can’t change after their first evaluation, because if they never change, and the initial evaluation comes out false, then the dependent work will remain stuck forever. A classic example of this pattern occurs when writing a precondition to be based against a fixed date - generically: "I want this step to wait to run only on May 1st, 2021 after 6:00pm" coded as follows: |
$[/javascript "$[/timestamp YYYY-MM-dd HH:mm]" > "2021-05-01 18:00" ]
Assuming this logic check is reached before the given time on May 1st, then the step will simply evaluate as FALSE, and will remain stuck. The date based expression contains no associated property to force a re-check of the logic. So this means there is no way for the expression to get a second chance to get re-evaluated when the 6:00pm time on the May 1st date has actually been reached.
Additional Questions about preconditions:
1) How should the dependent property be updated?
This will be based on your implementation. Some examples are as follows:
-
You launch a set of parallel steps where one of the steps is programmed to only start after a different step has first activated some value. Such a construct might be helpful in allowing sets of steps to cascade forward. This can permit some steps to run concurrently where a serial-only design, that doesn’t require any pre-conditions would result in the work taking much longer to complete.
-
You launch a set of concurrent jobs. Both perform some independent tasks but at some point, job2 needs to wait for something in job1 to be handled first, so a pre-condition can wait for job1 to have set a property that allows job2 to move forward.
-
You design a procedure where a step uses a precondition to wait on a property that is setup at run time but contains no value. The expectation is that some user must modify that property value to contain "1" or "TRUE" manually (using the UI or API) to allow the procedure to move forward. This essentially allows a procedure to emulates the concept of a "manual task" or gates inside pipeline definitions.
2) Could numerous steps be made to be waiting on 1 property to become TRUE?
A - As in the example in (1) above, you could create such a design, but you should consider the impact of your overall precondition design carefully. In particular, with an awareness to how many such jobs might ultimately be activated to run concurrently. The "review list" relies on a process that will scan regularly the set of dependent properties to identify when one has changed - so the more items you have on that list, the larger the performance impact might be for scanning this list. Therefore, limiting the number of waiting steps is always prudent.
3) Could the property I set my dependency against be a global property?
A - It’s quite common to see a process design work seamlessly when first crafted and tested, but months later, as production usage expands, this procedure might have significant numbers running concurrently. The idea of all of these jobs potentially waiting on some change to a global property seems unlikely to be what was first intended. So while nothing prevents you from referencing any property in the system, as your dependent property, having the dependency be based on a unique run-time property will typically be the better way to set your design.
4) If my step seems stuck - and I think it should have proceeded by this point, how can I verify what is going wrong with my precondition logic?
A - Unfortunately, the state of the precondition queue is not something that the log files keep records against, so this might become a tricky problem to resolve. If the logic is relatively straightforward, then perhaps just reviewing the current property value involved is all you need to do (use the UI or getProperty API). If the logic involves multiple properties, then this may take more effort.
A key lesson to take from this reality is that formulating overly complex precondition rules should be avoided.
Another lesson is that it is best to work out all your test case logic patterns before moving such designs into to your production system.
When first designing your logic, it may be beneficial to create a temporary "print" debugging step just prior to the step relying on the precondition. This "print" step simply needs to output the current values of any of the dependent properties, and ideally the evaluation of the same precondition logic. This won’t help you watch over the current values, but will show you how the step likely was evaluated at the time it was first ready to start. Such a print step is imperfect, of course, with a dynamic system, since what a property evaluates to at the time of the "print" step could still be different by the time the actual step with the precondition is evaluated.
This reinforces why creating a precondition that tries to work with values from the same run instantiation (instead of a global property) can help to limit risk for overlap or unexpected changes to the property value.