Jenkins Hangs on Workspaces Cleanup

Article ID:115001419832
2 minute readKnowledge base

Issue

"Computer.threadPoolForRemoting [#...]" ... waiting for monitor entry [...]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:49)
	- waiting to lock <...> (a com.thoughtworks.xstream.core.DefaultConverterLookup)
	at com.thoughtworks.xstream.XStream$1.lookupConverterForType(XStream.java:498)
	at ...
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
	at hudson.XmlFile.write(XmlFile.java:170)
	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.persist(AsyncResourceDisposer.java:175)
	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.access$300(AsyncResourceDisposer.java:65)
	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:262)
	at ...

Resolution

The Resource Disposer plugin has a known issue JENKINS-46460 that is causing massive contention on a single lock in XSTREAM.

When the issue is exposed:

  • the administrative monitor of the Resource Disposer is active. You can check for the details of the cleanup tasks that are hanging at $JENKINS_URL/administrativeMonitor/AsyncResourceDisposer.

  • the thread dump show AsyncResourceDisposer.persist stacktraces like the following:

"Computer.threadPoolForRemoting [#...]" ... waiting for monitor entry [...]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:49)
	- waiting to lock <...> (a com.thoughtworks.xstream.core.DefaultConverterLookup)
	at com.thoughtworks.xstream.XStream$1.lookupConverterForType(XStream.java:498)
	at ...
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
	at hudson.XmlFile.write(XmlFile.java:170)
	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.persist(AsyncResourceDisposer.java:175)
	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.access$300(AsyncResourceDisposer.java:65)
	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:262)
	at ...

How to address this ?

You can stop the cleanup task by hitting "Stop Tracking" on the entries listed in the administrative monitor page. This may help to reduce the load. A restart of the Client controller is usually needed to resolve the issue but only for a short term until the next hang.

Solution

The solution is to upgrade the Resource Disposer plugin to version 0.8 or later that contain the fix for JENKINS-46460.

Workaround(s)

Otherwise, a workaround is to disable Resource Disposer Plugin (and also the Workspace Cleanup plugin which uses this functionality). If you are using Pipeline, the Workspace Cleanup plugin step is not needed. You can replace the workspace cleanup with shell/batch delete commands or with the deleteDir step.

It is also preferred to use the features of the various SCMs plugins to clean untracked or modified files before updating. This avoids the need for an expensive full clone.