CloudBees Replication Parent 2542

As of 2.504.1.6 the /health endpoint (which should be used for Kubernetes probes going forward) in an HA controller would verify that the replica’s Hazelcast networking library was running, but not that it was actually useful for purposes of communicating with other replicas: for example, if port 5701 were blocked, it would be considered healthy, even though every replica thought itself to be in a cluster of size one. Now the health check tracks the set of running replicas via a directory in the shared $JENKINS_HOME, so if there is a split-brain scenario in Hazelcast, the controller will automatically scale itself down to one replica until the situation is resolved.

A Pipeline build starting just as a replica of an HA controller was in its shutdown sequence could have failed immediately with a stack trace of HazelcastInstanceNotActiveException related to FlowExecutionListStorage.register, because it was too late to add it to the list of running builds as recognized by other replicas. Now the warning is nonfatal.

Given a permanent agent with multiple executors running on windows, the path for extra nodes wasn’t computed properly and this was leading to errors preventing startup of these extra agents.