Required Data: Hang Issue On Linux Jenkins
This article describes how to collect the minimum required information for performance issues on a Jenkins environment so that it can be efficiently troubleshooted.
If the required data is bigger than 50 MB you will not be able to use ZenDesk to upload all the information. On this case we would like to encourage you to use our upload service in order to attach all the required information.
Environment
-
Operating System: Linux
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
-
CloudBees CI (CloudBees Core) on traditional platforms - Client controller
-
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
Automatic Data collection
This is the preferred method if you are using a product supported by the cbsupport
CLI.
Current products supporting collecting performance data are:
Steps to follow are:
-
Install and configure
cbsupport
following Using cbsupport CLI to collect the requested data -
Run
cbsupport required-data performance
-
Collect the archive generated in the working directory of cbsupport and attach it to the ticket using our upload service
Manual Data collection - Required Data check list
Ideally, your environment is configured as documented in Prepare CloudBees CI for Support.
-
Support bundle
-
Output of the script
collectPerformanceData.sh
-
GC log file for review
-
Optional (if an I/O issue is suspected): output of the script listDProcessesNativeStacks.sh
Support bundle
A support bundle from the problematic instance ideally when the issue is happening or in the worst case right after the issue is exposed. Please, follow the KB below in case you don’t know how to generate a support bundle.
Please, enable at minimum the following checks: System properties
, Controller Log Recorders
, Garbage Collection Logs
, Slow Request Records
, Out Of Memory Errors
, Controller Heap Histogram
, Deadlock Records
, Thread dumps of running Pipeline builds
and Thread dumps
.
Find more details about the Support Bundle in What is a support-bundle and how to create one.
Output of the script collectPerformanceData.sh
Depending on the type of products you have different approaches:
CloudBees CI on traditional platforms
This process is also valid for CloudBees Jenkins Platform (Older product) and Jenkins LTS.
Steps
1. - Check you have the required tools to run the script included in the $PATH
The script must run as the same user as Jenkins. In the next steps we will consider this user to be named jenkins. The following checks must be performed with this users.
The collectPerformanceData.sh
script collects thread dumps using either the jstack
, jcmd
or jattach
commands.
There are 2 options here:
-
either there is a JDK installed on the machine, in which case you need to make sure that either
jstack
orjcmd
are available either in your PATH or in the path pointed to by theJAVA_HOME
variable. -
or the machine only has a JRE installed, in which case you will need to download jattach from here. Once
jattach
is downloaded, make sure it is executable for the user jenkins and either add it to your PATH or make the variableJATTACH_HOME
to the directory containing it.
The script works optimally when the following optional commands are available top
, vmstat
, netstat
, nfsiostat
, nfsstat
and iostat
.
2. - Download the collectPerformanceData.sh script.
3. - Make the script executable
The collectPerformanceData.sh script will need to be made executable by running: chmod +x collectPerformanceData.sh
.
4. - Get the $JENKINS_USER
and $JENKINS_PID
You can retrieve the PID running ps ax | grep -i java | grep -i -e jenkins -e cloudbees | grep war | awk '{print $1}'
5. - Check that the jenkins user has permissions in the directory where collectPerformanceData.sh
is running
The collectPerformanceData.sh
script will generate data in this directory.
Alternatively you can make the script generate its output elsewhere by setting up the JENKINS_HANG_OUTPUT_DIR
variable.
In this case, the jenkins user should have rights to write in JENKINS_HANG_OUTPUT_DIR
.
6. - If you are experiencing a hang or a poor performance issue please execute the script while the issue is occurring:
Run the collectPerformanceData.sh
script as the jenkins user by running:
sudo -u jenkins collectPerformanceData.sh $JENKINS_PID
7. - Collect the archive generated and attach it to the ticket using our upload service
Example
Running ps ax | grep -i java | grep -i -e jenkins -e cloudbees | grep war | awk '{print $1}'
we get:
50946
We can now run the script, the user running Jenkins being jenkins
sudo -u jenkins sh collectPerformanceData.sh 50946
You can run the script without any arguments to validate your setup.
sudo -u jenkins sh collectPerformanceData.sh
If the collectPerformanceData.sh script does not run as expected please upload the output the script generates from the terminal window for review.
|
CloudBees CI on modern platforms
Steps
In addition to the process described for traditional platform, take into account the following considerations CloudBees CI on Modern: Performance Issues - High CPU / Blocked Threads.
GC log file for review.
If you followed Prepare CloudBees CI for Support then the gc logs file should be at -Xlog:gc[...]:file=<some-log-file>:[...]
for JDK 11+ or -Xloggc:<some-log-file>
for JDK 8: