Deadlock on logging.StreamHandler

Article ID:360031146092
2 minute readKnowledge base

Issue

On startup, receiving deadlocks that prevent Jenkins from fully starting. Grabbing a thread dump will show the stack trace for these deadlocks. Looking at the stacktrace, it might appear to be caused by a plugin. Some indicative behavior also includes attempting to start Jenkins multiple times and getting different stack traces for the deadlock each time.

Stack Trace might look similar to:

"com.sonyericsson.hudson.plugins.gerrit.trigger.GerritProjectListUpdater for EDC-Gerrrit Thread":
at java.util.logging.StreamHandler.publish(StreamHandler.java:206)
- waiting to lock <0x00000005dfe414d0> (a java.util.logging.StreamHandler)
at com.cloudbees.jenkins.support.impl.JenkinsLogs$LogFile.publish(JenkinsLogs.java:224)
at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:265)
at java.util.logging.Logger.log(Logger.java:738)
at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:582)
at org.slf4j.impl.JDK14LoggerAdapter.info(JDK14LoggerAdapter.java:320)
at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritProjectListUpdater.tryLoadProjectList(GerritProjectListUpdater.java:226)
at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritProjectListUpdater.run(GerritProjectListUpdater.java:158)
"GrapeHack.hack":

The key line to note in the stack trace is this:

waiting to lock <0x00000005dfe414d0> (a java.util.logging.StreamHandler)

This issue is stemming from a bug with locked logging resources.

Environment

Workaround

In order to clear out the deadlock here, look at what custom loggers are defined in this Jenkins instance.

With the system down you can locate these by navigating to $JENKINS_HOME/log on the filesystem. Each custom logger is saved there as an xml file.

These issues are stemming from an issue with higher logging levels in a custom log. To workaround the issue and get the Jenkins instance back up and running, remove high level loggers or all loggers defined in $JENKINS_HOME/log and restart Jenkins

Tested product/plugin versions

  • CloudBees Core on Traditional Platforms version 2.176.2