Issue
-
The Jenkins controller thread dump show lot of threads like the following:
"QueueSubTaskMetrics [#XXX]" #XXXXX daemon prio=5 os_prio=0 tid=0x00001a1234567890 nid=0xa46e waiting on condition [0x00001a0987654321] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000q2w3e4r5> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Environment
-
CloudBees CI (CloudBees Core) < 2.204.1.3
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller < 2.204.1.3
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center < 2.204.1.3
-
CloudBees CI (CloudBees Core) on traditional platforms - Client controller < 2.204.1.3
-
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center < 2.204.1.3
-
CloudBees Jenkins Platform - Client controller < 2.204.1.3
-
CloudBees Jenkins Platform - Operations Center < 2.204.1.3
-
CloudBees Jenkins Distribution < 2.204.1.3
-
Jenkins LTS < 2.204.1
-
Metrics Plugin >= 4.0.2.0
Explanation
The Metrics plugin monitors the Queue asynchronously. It is leaking QueueSubTaskMetrics threads when a SubTask from the build queue fails badly - in such a way that the task is never updated. In such cases the QueueSubTaskMetrics thread wait forever for a event that will never occur.
The controller needs to be restarted to clean those threads.
The root cause of this issue is actually a bug in Jenkins core that is not updating the SubTask consistently in case of premature failure captured by JENKINS-59793.