Shared agents stuck in unknown state and jobs queue indefinitely after upgrade

Last Reviewed:2025-11-06()
3 minute readKnowledge base

Issue

After upgrading operations center and client controller to version 2.504.2.5 with OpenJDK 17, shared agents fail to lease to controllers despite appearing online and idle in the Jenkins UI. Jobs queue continuously with "no nodes available with label X" messages even though matching agents are shown as available in operations center.

This issue primarily affects shared agents configured with 3 or more executors.

Operations Center logs show repeated provisioning exceptions:

Sep 29, 2026 1:35:59 PM WARNING com.cloudbees.opscenter.client.cloud.CloudImpl$1 call
Exception creating node LeaseId{uuid='f64e2b17-4a85-41da-8e6e-74ddd21831ff'}
com.cloudbees.opscenter.provisioning.ProvisioningException: Cannot provision a node called l04_build4 when to a Jenkins instance with an existing node of the same name
                at PluginClassLoader for operations-center-cloud//com.cloudbees.opscenter.client.cloud.CloudImpl$1.call(CloudImpl.java:168)
                at PluginClassLoader for operations-center-cloud//com.cloudbees.opscenter.client.cloud.CloudImpl$1.call(CloudImpl.java:151)
                at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
                at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
                at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
                at java.base/java.lang.Thread.run(Thread.java:840)

Controller logs show related connection-tracking errors on Windows agents:

Refusing headers from remote: Unknown client name: XXXXXXX

Some agents still running an older Java version report:

UnsupportedClassVersionError: ChannelPinger$SetUpRemotePing has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0

Additional symptoms observed:

  • Agents become permanently "zombie" records that can never be leased again under the same name.

  • Executor count mismatches between configuration and the UI.

  • Controller restarts temporarily clear some, but not all, instances of the problem.

  • Job queue depth reaches far above normal levels (normally 1-3 queued jobs; observed 32-43+ queued items, with up to 81% blocked by this defect on a single controller).

This prevents jobs from executing, severely impacting build throughput.

Environment

  • CloudBees CI on modern cloud platforms - managed controller, operations center and client controller, version 2.504.2.5

  • Shared-agent infrastructure leased from operations center to multiple client controllers

  • OpenJDK 17 (mandatory starting with core version 2.479.1.3)

  • Shared agents configured with 3 or more executors

Related Issue(s)

Resolution

This issue is fixed in CloudBees CI version 2.528.2.34846 and later.

  1. Plan an upgrade of operations center and all client controller to version 2.528.2.34846 or 2.541.3.36069 (or later).

  2. Upgrade operations center first, ensuring it remains at the same or a higher version than all client controllers.

  3. Upgrade each client controller in order, verifying agent connectivity after each upgrade.

  4. After the upgrade completes, verify that agents lease normally and that the job queue clears.

Before upgrading, correct the Job Config History plugin configuration described below. Without this change, lease synchronization issues can recur on high-job-count controllers even after upgrading.

  1. On operations center, navigate to Manage Jenkins  Job Config History and uncheck Do not save duplicate history.

  2. Add XmlSlaveManifestImpl to the Job Config History exclude pattern.

  3. Apply the same Job Config History configuration changes on each client controller.

Take a backup of operations center and each client controller before performing the upgrade.

The fix is included in CloudBees CI version 2.528.2.34846 and later.

Workaround

Use these workarounds only until the upgrade in Resolution can be completed. None of these workarounds fix the underlying race condition; they only mitigate symptoms and must be repeated or maintained until the upgrade completes.

Option 1 - Reduce executor count:

  1. In operations center, open the configuration for each affected shared agent that has 3 or more executors.

  2. Reduce the executor count to 2 or fewer.

  3. Restart or reconnect the agent to apply the new executor count.

Reducing executor count from 5 to 2 results in approximately 60% capacity loss per agent.

Option 2 - Split into multiple agents:

  1. Create new shared agent configurations under different names (for example, build1-2, build2-2, build3-2) with 2 executors each.

  2. Assign the same labels as the original agents so that controller jobs can still route to them.

  3. Register the new agent names with operations center and confirm they receive leases.

Option 3 - Clean up zombie agent records:

  1. On each client controller, navigate to Manage Jenkins  Script Console.

  2. Run the following script to remove stale node records that have no corresponding Computer object:

    def j = Jenkins.get()
def removed = []
j.getNodes().each { n ->
    if (n.toComputer() == null) {
        removed << n.name
        j.removeNode(n)
    }
}
println "Removed ${removed.size()} stale node(s): ${removed}"
    ▼

  3. Repeat this cleanup periodically (for example, every 5 minutes via a scheduled pipeline job) until the upgrade is completed, as new zombie records continue to accumulate.

Additional step - Release stuck leases:

  1. In operations center, open the agent leasing display.

  2. Identify any agents shown as leased but idle.

  3. Click Force release on each stuck lease.

This workaround is temporary and must be maintained until the upgrade described in the Resolution section is completed.

Tested product/plugin versions

References

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.