Required Data: Jenkins: High CPU On Linux

Article ID:229370407
3 minute readKnowledge base

Issue

A specific job or your Jenkins environment is experiencing high CPU usage.

Required Data: High CPU On Linux

This article describes how to collect the minimum required information for high CPU issues on a Jenkins environment to aid in troubleshooting.

If the required data is bigger than 50 MB you will not be able to use ZenDesk to upload all the information. In this case, please use our upload service in order to attach all the required information. The upload service works best in Chrome or Firefox.

Environment

Automatic Data collection

This is the preferred method if you are using a product supported by the cbsupport CLI. Current products supporting collecting performance data are:

Steps to follow are:

  1. Set up cbsupport

  2. Run cbsupport required-data performance

  3. Collect the archive generated in the working directory of cbsupport and attach it to the ticket using our upload service

Manual Data collection - Required Data check list

Ideally, your environment is configured as documented in Prepare CloudBees CI for Support.

  • Support bundle

  • Output of the script collectPerformanceData.sh

  • GC log file for review

Support Bundle

Please capture a support bundle from the problem instance, ideally when the issue is happening, or else right after the issue is exposed. This knowledge base article describes how to create a support bundle.

Please enable at minimum the following checks: System properties, controller Log Recorders, Garbage Collection Logs, Slow Request Records, Out Of Memory Errors, controller Heap Histogram, Deadlock Records, Thread dumps of running Pipeline builds, Thread dumps and Items Content.

Output of the script collectPerformanceData.sh

Pre-requisite: Download the collectPerformanceData.sh script.

If the collectPerformanceData.sh script does not run as expected please upload into the support ticket the output the script generates from the terminal window for review.

Traditional platform

1. Check you have the required tools to run the script included in the $PATH

See CloudBees Jenkins Platform Supported Java Versions and ensure that you are using a supported Java environment.

The collectPerformanceData.sh script collects thread dumps using the jstack command. It also needs to be able to run top, vmstat, netstat, nfsiostat, nfsstat and iostat. Please make sure that the user Jenkins is running as can execute all of these commands.

2. Make the script executable by running: chmod +x collectPerformanceData.sh.

3. Determine the $JENKINS_USER and $JENKINS_PID

Run ps -ef | grep jenkins (or jps | grep jenkins). You will see output similar to:

jenkins 12345 17347 0 Mar17 ? 00:00:17 /usr/bin/java -jar jenkins.war

The first two columns show the user and process ID. In this case, $JENKINS_USER is jenkins and $JENKINS_PID is 12345. It is best to use ps to determine these values, rather than looking at the Jenkins service PID file, because on some systems the PID file contains the process ID of the daemon/service process, rather than Jenkins itself.

4. Check that the Jenkins user can write to the same folder where collectPerformanceData.sh is running. The script will generate data in that directory.

5. If you are experiencing a high CPU issue please execute the script while the issue is occurring. Run the collectPerformanceData.sh script as the same user that starts Jenkins by running:

sudo -u $JENKINS_USER sh collectPerformanceData.sh $JENKINS_PID 300 5

Example

From ps aux | grep jenkins

avahi 1115 0.0 0.0 32344 1328 ? Ss 2017 0:47 avahi-daemon: running [jenkins.local]
exampleUser 3045 236 16.2 10286700 2645660 ? Ssl Jan02 6592:33 /etc/alternatives/java -Dhudson.slaves.WorkspaceList=- -Djava.awt .headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/ var/cache/jenkins/war --httpPort=8086 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
root 10366 0.0 0.0 112652 964 pts/0 S+ 03:01 0:00 grep --color=auto jenkins

Then

sudo -u exampleUser sh collectPerformanceData.sh 3045 300 5

The output from this script can be understood by reviewing What is collectPerformanceData.sh and how does it help?.

Modern platform

As explained on the Troubleshooting CloudBees CI on Kubernetes docs page into the Performance issues - high CPU/blocked threads section

GC log file for review

If you followed Prepare CloudBees CI for Support then the gc logs file should be under -Xloggc:$path/gc.log