Required Data: High CPU On Linux
This article describes how to collect the minimum required information for high CPU issues on a Jenkins environment to aid in troubleshooting.
If the required data is bigger than 50 MB you will not be able to use ZenDesk to upload all the information. In this case, please use our upload service in order to attach all the required information. The upload service works best in Chrome or Firefox.
Environment
-
Operating System: Linux
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
-
CloudBees CI (CloudBees Core) on traditional platforms - Client controller
-
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
Automatic Data collection
This is the preferred method if you are using a product supported by the cbsupport
CLI.
Current products supporting collecting performance data are:
Steps to follow are:
-
Collect the archive generated in the working directory of cbsupport and attach it to the ticket using our upload service
Manual Data collection - Required Data check list
Ideally, your environment is configured as documented in Prepare Jenkins For Support.
-
Support bundle
-
Output of the script
collectPerformanceData.sh
-
GC log file for review
Support Bundle
Please capture a support bundle from the problem instance, ideally when the issue is happening, or else right after the issue is exposed. This knowledge base article describes how to create a support bundle.
Please enable at minimum the following checks: System properties
, controller Log Recorders
, Garbage Collection Logs
, Slow Request Records
, Out Of Memory Errors
, controller Heap Histogram
, Deadlock Records
, Thread dumps of running Pipeline builds
, Thread dumps
and Items Content
.
Output of the script collectPerformanceData.sh
Pre-requisite: Download the collectPerformanceData.sh script.
If the collectPerformanceData.sh
script does not run as expected please upload into the support ticket the output the script generates from the terminal window for review.
Traditional platform
1. Check you have the required tools to run the script included in the $PATH
See CloudBees Jenkins Platform Supported Java Versions and ensure that you are using a supported Java environment.
The collectPerformanceData.sh
script collects thread dumps using the jstack
command. It also needs to be able to run top
, vmstat
, netstat
, nfsiostat
, nfsstat
and iostat
. Please make sure that the user Jenkins is running as can execute all of these commands.
2. Make the script executable by running: chmod +x collectPerformanceData.sh
.
3. Determine the $JENKINS_USER
and $JENKINS_PID
Run ps -ef | grep jenkins
(or jps | grep jenkins
). You will see output similar to:
jenkins 12345 17347 0 Mar17 ? 00:00:17 /usr/bin/java -jar jenkins.war
The first two columns show the user and process ID. In this case, $JENKINS_USER
is jenkins
and $JENKINS_PID
is 12345
. It is best to use ps
to determine these values, rather than looking at the Jenkins service PID file, because on some systems the PID file contains the process ID of the daemon/service process, rather than Jenkins itself.
4. Check that the Jenkins user can write to the same folder where collectPerformanceData.sh
is running. The script will generate data in that directory.
5. If you are experiencing a high CPU issue please execute the script while the issue is occurring. Run the collectPerformanceData.sh
script as the same user that starts Jenkins by running:
sudo -u $JENKINS_USER sh collectPerformanceData.sh $JENKINS_PID 300 5
Example
From ps aux | grep jenkins
avahi 1115 0.0 0.0 32344 1328 ? Ss 2017 0:47 avahi-daemon: running [jenkins.local]
exampleUser 3045 236 16.2 10286700 2645660 ? Ssl Jan02 6592:33 /etc/alternatives/java -Dhudson.slaves.WorkspaceList=- -Djava.awt .headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/ var/cache/jenkins/war --httpPort=8086 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
root 10366 0.0 0.0 112652 964 pts/0 S+ 03:01 0:00 grep --color=auto jenkins
Then
sudo -u exampleUser sh collectPerformanceData.sh 3045 300 5
The output from this script can be understood by reviewing What is collectPerformanceData.sh and how does it help?.
Modern platform
As explained on the Troubleshooting CloudBees CI on Kubernetes docs page into the Performance issues - high CPU/blocked threads section
GC log file for review
If you followed Prepare Jenkins For Support then the gc logs file should be under -Xloggc:$path/gc.log