Non-concurrent Pipeline builds remain stuck in the queue after an HA controller replica is terminated while loading the build definition

Last Reviewed:2026-10-19()
1 minute readKnowledge base

Issue

On a High Availability (HA) controller, you observe that a non-concurrent Pipeline build has become permanently stuck. The build shows as running, but makes no progress, and the build queue blocks every subsequent build of the same Pipeline with a message similar to:

In queue for: 15 min
Is blocked: true
Why in queue: Still running on <replica-pod-name>

The replica pod named in the message may no longer exist. The Build History widget can also show a duplicate entry for the same build number, with one entry pointing at the non-existent replica.

Inspecting the stuck build’s build.xml shows completed set to true and an OwnerAction pointing at a replica, but no <execution> element, for example:

<flow-build plugin="workflow-job@1571.1580.v18e46842c125">
  <actions>
    <loadBalancer plugin="cloudbees-replication@3329">
      <id>3539321718587189321</id>
      <initialHost>instance-7cfb6c65d8-hv52t</initialHost>
    </loadBalancer>
    <com.cloudbees.jenkins.plugins.replication.builds.OwnerAction plugin="cloudbees-replication@3329">
      <owners>
        <com.cloudbees.jenkins.plugins.replication.builds.OwnerAction_-Owner>
          <timestamp>1787805859652</timestamp>
          <name>instance-7cfb6c65d8-4h68t</name>
        </com.cloudbees.jenkins.plugins.replication.builds.OwnerAction_-Owner>
      </owners>
    </com.cloudbees.jenkins.plugins.replication.builds.OwnerAction>
  </actions>
  <queueId>3539321718587189321</queueId>
  <timestamp>1787805859531</timestamp>
  <startTime>1787805859559</startTime>
  <duration>0</duration>
  <completed>true</completed>
  <checkouts class="hudson.util.PersistedList"/>
</flow-build>
▼

The absence of an <execution> element shows that the Groovy program never started, even though the build is recorded as completed. This prevents all later builds of the same non-concurrent Pipeline from starting, since the controller believes a build is still in progress on a replica that is no longer running it.

Environment

Related Issue(s)

Explanation

A Pipeline build goes through an initial phase in which its definition is loaded, for example retrieving the Jenkinsfile from SCM or cloning a Pipeline library, before the Groovy logic is interpreted and the program state is recorded.

On an HA (active/active) controller, every replica records build ownership in distributed Hazelcast maps, such as Adoption.owners and ReplicatedHistoryWidget.builds, so that any replica can resume a build if the replica that owns it goes away.

If the owning replica is terminated while the build definition is still loading, no program state has been recorded yet, so the other replicas cannot resume the build. The build remains recorded as running without making progress, and the corrupted ownership entry also blocks later builds of the same non-concurrent Pipeline.

A replica can be terminated during this phase for several reasons, including a Kubernetes pod eviction (for example, due to ephemeral storage exhaustion), a node drain, or a manual restart.

Resolution

Upgrade to CloudBees CI version 2.580.1.38223 or newer to resolve this issue.

Workaround

If you cannot upgrade immediately:

  1. Identify the replica pod named in the Still running on <replica-pod-name> queue message.

  2. Restart that replica, or perform a rolling restart of the managed controller, to reset the Hazelcast state.

  3. Re-trigger the blocked builds of the affected Pipeline.

This workaround only clears the current stuck build. It does not prevent the issue from recurring until you upgrade to a fixed version.

References

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.