KBEC-00457 - Cluster time skew causes: java.lang.IllegalArgumentException: durationMillis must not be negative

Article ID:360050360692
1 minute readKnowledge base

Issue

If running CloudBees CD (Flow) in clustered mode (more than one CD server), and the system clocks have a small time skew betwwen them, possibly due to them using different NTP servers, or them not using a NTP server, you will encounter the error:

[InternalError]: java.lang.IllegalArgumentException: durationMillis must not be negative
Details:
java.lang.IllegalArgumentException: durationMillis must not be negative
at org.apache.commons.lang3.Validate.inclusiveBetween(Validate.java:1055)
at org.apache.commons.lang3.time.DurationFormatUtils.formatDuration(DurationFormatUtils.java:130)
at org.apache.commons.lang3.time.DurationFormatUtils.formatDuration(DurationFormatUtils.java:113)
at org.apache.commons.lang3.time.DurationFormatUtils.formatDurationWords(DurationFormatUtils.java:180)
at com.electriccloud.domain.JobStepImpl.getDuration(JobStepImpl.java:974)
...

The bug will only be exposed if you have clock skew between the server nodes of your CD cluster, and if you have job steps that take fewer milliseconds to complete than the clock skew between your nodes.

To confirm you have clock skew between your CloudBees CD cluster nodes, login to one of the nodes, and run:

sudo ntpdate -qu ${HOSTNAME_OR_IP_OF_OTHER_HA_NODE}

This ntpdate command will print out the time offset between the current server and the host you provided to the command.

Environment

Resolution

Setup NTP on all CD server nodes, to make sure there is no clock skew.

Tested product/plugin versions