Required Data: High CPU On Linux
This article describes how to collect the minimum required information for high CPU issues on a Jenkins environment to aid in troubleshooting.
If the required data is bigger than 50 MB you will not be able to use ZenDesk to upload all the information. In this case, please use our upload service in order to attach all the required information. The upload service works best in Chrome or Firefox.
Operating System: Linux
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
CloudBees CI (CloudBees Core) on traditional platforms - Client controller
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
Automatic Data collection
This is the preferred method if you are using a product supported by the
Current products supporting collecting performance data are:
Steps to follow are:
Collect the archive generated in the working directory of cbsupport and attach it to the ticket using our upload service
Manual Data collection - Required Data check list
Ideally, your environment is configured as documented in Prepare Jenkins For Support.
Output of the script
GC log file for review
Please capture a support bundle from the problem instance, ideally when the issue is happening, or else right after the issue is exposed. This knowledge base article describes how to create a support bundle.
Please enable at minimum the following checks:
controller Log Recorders,
Garbage Collection Logs,
Slow Request Records,
Out Of Memory Errors,
controller Heap Histogram,
Thread dumps of running Pipeline builds,
Thread dumps and
Output of the script collectPerformanceData.sh
Pre-requisite: Download the collectPerformanceData.sh script.
collectPerformanceData.sh script does not run as expected please upload into the support ticket the output the script generates from the terminal window for review.
1. Check you have the required tools to run the script included in the
See CloudBees Jenkins Platform Supported Java Versions and ensure that you are using a supported Java environment.
collectPerformanceData.sh script collects thread dumps using the
jstack command. It also needs to be able to run
iostat. Please make sure that the user Jenkins is running as can execute all of these commands.
2. Make the script executable by running:
chmod +x collectPerformanceData.sh.
3. Determine the
ps -ef | grep jenkins (or
jps | grep jenkins). You will see output similar to:
jenkins 12345 17347 0 Mar17 ? 00:00:17 /usr/bin/java -jar jenkins.war
The first two columns show the user and process ID. In this case,
12345. It is best to use
ps to determine these values, rather than looking at the Jenkins service PID file, because on some systems the PID file contains the process ID of the daemon/service process, rather than Jenkins itself.
4. Check that the Jenkins user can write to the same folder where
collectPerformanceData.sh is running. The script will generate data in that directory.
5. If you are experiencing a high CPU issue please execute the script while the issue is occurring. Run the
collectPerformanceData.sh script as the same user that starts Jenkins by running:
sudo -u $JENKINS_USER sh collectPerformanceData.sh $JENKINS_PID 300 5
ps aux | grep jenkins
avahi 1115 0.0 0.0 32344 1328 ? Ss 2017 0:47 avahi-daemon: running [jenkins.local] exampleUser 3045 236 16.2 10286700 2645660 ? Ssl Jan02 6592:33 /etc/alternatives/java -Dhudson.slaves.WorkspaceList=- -Djava.awt .headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/ var/cache/jenkins/war --httpPort=8086 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20 root 10366 0.0 0.0 112652 964 pts/0 S+ 03:01 0:00 grep --color=auto jenkins
sudo -u exampleUser sh collectPerformanceData.sh 3045 300 5
The output from this script can be understood by reviewing What is collectPerformanceData.sh and how does it help?.
As explained on the Troubleshooting CloudBees CI on Kubernetes docs page into the Performance issues - high CPU/blocked threads section
GC log file for review
If you followed Prepare Jenkins For Support then the gc logs file should be under