Jenkins hangs due to Groovy "ClassInfo" deadlock

Article ID:115001505172
2 minute readKnowledge base

Issue

  • Queue is filling up. Jenkins does not process jobs in the queue

  • Nodes are stuck

  • Jenkins sometimes hangs and the thread dump show several (BLOCKED) threads similar to:

"Executor [#...] for <node> : executing <job> [#...]" ... waiting for monitor entry [...]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.codehaus.groovy.reflection.ClassInfo$GlobalClassSet.add(ClassInfo.java:477)
    - waiting to lock <...> (a org.codehaus.groovy.util.ManagedLinkedList)
    at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:83)
    at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:79)
    at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.<init>(GroovyClassValuePreJava7.java:37)
    [...]
    at hudson.model.Executor.run(Executor.java:405)

or

"Executor [#...] for <node> : executing <job> [#...]" ... state=BLOCKED cpu=100%
    - waiting to lock <...> (a org.codehaus.groovy.util.ManagedLinkedList)
      owned by "Executor [#...] for <node> : executing <job> [#...]" ...
    at org.codehaus.groovy.reflection.ClassInfo$GlobalClassSet.add(ClassInfo.java:477)
    at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:83)
    at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:79)
    at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.<init>(GroovyClassValuePreJava7.java:37)

Environment

  • CloudBees Jenkins Enterprise - Managed controller (CJEMM)

  • CloudBees Jenkins Platform - Client controller (CJPCM)

  • CloudBees Jenkins Team (CJT)

  • Jenkins LTS 2.60.X

Related Issues

Resolution

Jenkins Core ships with an embedded distribution of Groovy. Jenkins 2.60.x ships with Groovy 2.4.8 and this version has a known deadlock issues that can causes serious lock contention in Jenkins. It is captured by GROOVY-8067 in the Groovy project and JENKINS-43197 in the Jenkins project.

If an instance is impacted by this issue, a thread dump taken when the issue would show several threads BLOCKED on the method org.codehaus.groovy.reflection.ClassInfo$GlobalClassSet.add:

"Executor [#...] for <node> : executing <job> [#...]" ... waiting for monitor entry [...]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.codehaus.groovy.reflection.ClassInfo$GlobalClassSet.add(ClassInfo.java:477)
    - waiting to lock <...> (a org.codehaus.groovy.util.ManagedLinkedList)
    at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:83)
    at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:79)
    at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.<init>(GroovyClassValuePreJava7.java:37)
    [...]
    at hudson.model.Executor.run(Executor.java:405)

How to address this ?

A restart of the Client controller is usually needed to resolve the issue but only for a short term until the next hang.

The issue is most likely related to expensive Groovy computation on the controller. The stacktrace of the BLOCKED threads can help to identify the <job> that is (are) causing the memory leak. If you are using Pipeline, avoid Pipeline XML or JSON parsing using Groovy’s XmlSlurper and JsonSlurper as explained in the Pipeline best practices.

In any case, the solution is to upgrade to Jenkins 2.73.x or later. This deadlock has been fixed in Groovy 2.4.9 and this fix will be included in Jenkins LTS 2.73 (that ships with Groovy 2.4.11).

(Deprecated) Workaround

JENKINS-43197 mentions a workaround that is to add the system property -Dgroovy.use.classvalue=true on Jenkins startup to re-enable ClassValue. This is deprecated and we strongly advise to not do this. ClassValue was disabled by default to solve major memory leaks - see GROOVY-7591. Enabling ClassValue would replace a groovy memory leak by another one.