Workspace CleanUp threads may cause performance issues

Article ID:360047019911
1 minute readKnowledge base

Issue

  • Jenkins suffer slowness and performance issues

  • branch-api version is lower than 2.5.6

  • In the thread dumps you can observe BLOCKED threads like

"Computer.threadPoolForRemoting [#15979]" #398547 daemon prio=5 os_prio=0 tid=0x00007f0509fcf000 nid=0x1e982 waiting for monitor entry [0x00007f0246221000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at jenkins.branch.WorkspaceLocatorImpl.locate(WorkspaceLocatorImpl.java:158)
	- waiting to lock <0x00000003c0677358> (a hudson.slaves.DumbSlave)
	at jenkins.branch.WorkspaceLocatorImpl.locate(WorkspaceLocatorImpl.java:129)
	at jenkins.branch.WorkspaceLocatorImpl.access$100(WorkspaceLocatorImpl.java:80)
	at jenkins.branch.WorkspaceLocatorImpl$Deleter$CleanupTask.run(WorkspaceLocatorImpl.java:402)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- <0x000000049de00a48> (a java.util.concurrent.ThreadPoolExecutor$Worker)

blocked waiting for the "Workspace clean-up thread"

"Workspace clean-up thread" #220769 daemon prio=5 os_prio=0 tid=0x00007f054cb47000 nid=0x1414 in Object.wait() [0x00007f046e386000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:175)
	- locked <0x000000048d215470> (a [B)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	- locked <0x000000048d227520> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.readLine(BufferedReader.java:324)
	- locked <0x000000048d227520> (a java.io.InputStreamReader)
	at java.io.BufferedReader.readLine(BufferedReader.java:389)
	at jenkins.branch.WorkspaceLocatorImpl.load(WorkspaceLocatorImpl.java:222)
	at jenkins.branch.WorkspaceLocatorImpl.locate(WorkspaceLocatorImpl.java:159)
	- locked <0x00000003c0677358> (a hudson.slaves.DumbSlave)
	at jenkins.branch.WorkspaceLocatorImpl.locate(WorkspaceLocatorImpl.java:129)
	at jenkins.branch.WorkspaceLocatorImpl.locate(WorkspaceLocatorImpl.java:125)
	at hudson.model.Slave.getWorkspaceFor(Slave.java:335)
	at hudson.model.WorkspaceCleanupThread.execute(WorkspaceCleanupThread.java:78)
	at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:101)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

Environment

Explanation

Thread contention with the WorkspaceLocatorImpl can cause Jenkins to hang. Large amounts of workspace cleanup threads, such as from deleting large Multibranch/Organization items, can cause this behaviour.

Other causes are possible like a filesystem poor performance.

Resolution

Upgrade Branch API plugin to version 2.5.6 or higher.

References

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.