Required Data: Thread Deadlock Issue

Article ID:360027928272
2 minute readKnowledge base

Issue

  • Jenkins hangs (may be anytime, including startup)

  • Jenkins is not responsive or coming up, the process is running and Jenkins logs are stale

Required Data: Hang Issue On Linux Jenkins

This article describes how to collect the minimum required information for deadlock issues on a Jenkins environment so that it can be efficiently troubleshooted.

If the required data is bigger than 50 MB you will not be able to use ZenDesk to upload all the information. On this case we would like to encourage you to use our upload service in order to attach all the required information.

Environment

Required Data check list

Ideally, your environment is configured as documented in Prepare CloudBees CI for Support.

  • Output of the script collectPerformanceData.sh

  • Support bundle

  • List of File Descriptors

  • Optional (if an I/O issue is suspected): output of the script listDProcessesNativeStacks.sh

Output of the script collectPerformanceData.sh

Thread dumps are the best way to pinpoint the root cause of this issue.

See Required Data: Jenkins Hang Issue On Linux KB.

Support bundle

A support bundle from the problematic instance ideally when the issue is happening or in the worst case right after the issue is exposed. Please, follow the KB below in case you don’t know how to generate a support bundle.

Please, enable at minimum the following checks: System properties, controller Log Recorders, Garbage Collection Logs, Slow Request Records, Out Of Memory Errors, controller Heap Histogram, Deadlock Records, Thread dumps of running Pipeline builds and Thread dumps.

List of File Descriptors

The list of file descriptor hold by the Jenkins process.

Find the PID of the Jenkins process. For example the following output of ps aux | grep jenkins show that Jenkins PID is 2487:

[ec2-user@ip-10-240-0-112 ~]$ ps -ef | grep jenkins
ec2-user  2487  2435  1 May07 ?        00:26:30 java -Duser.home=/var/jenkins_home -Xmx2150m -Xms2150m -jar /usr/share/jenkins/jenkins.war --webroot=/tmp/jenkins/war --pluginroot=/tmp/jenkins/plugins
ec2-user 14540 13480  0 03:20 pts/0    00:00:00 grep --color=auto jenkins

Collect the list of file descriptor of that process with lsof (or sudo lsof if sudo privileges are needed):

lsof -a -p $JENKINS_PID > /tmp/file-descriptors.txt

If lsof is not available on the host, you may use the following command instead:

ls -l /proc/$JENKINS_PID/fd/ > /tmp/file-descriptors.txt

Collect the file produced /tmp/file-descriptors.txt

Output of the script listDProcessesNativeStacks.sh

See the What is listDProcessesNativeStacks.sh and how does it help? KB.