Environment
These best practices are for the following products and should not be deployed for CI-Modern
Note
Please be aware that if Jenkins runs under control of systemd, then defining the ulimits as shown below within /etc/security/limits.conf
has no effect on Jenkins. Parameters like LimitNoFile
and LimitNoProc
must be set within the systemd service unit file.
Resolution
The following settings are known Best Practices, and should be used in your Jenkins environment to allow for easier support in the event of an issue.
A. Java Configuration
JVM Administration and Tuning is necessary for any Enterprise Java Application Deployment. As Jenkins is an Enterprise Java Application, the following arguments are considered to be best practices, based on historical data analysis of Garbage Collection Logs, Thread Dump Analysis, and Heap Dump Analysis.
In the last several years with Java 1.8.0, fine-tuning Garbage Collection settings was considered a common practice, but ultimately masks underlying problems within the JDK. Because of this, it was also common practice to allocate more RAM and CPU resources to the JVM to keep it running smoothly. While this will help in the short term, it still masks underlying problems. These methodologies are described in-depth in this article.
In today’s methodologies, it is more important to ensure you are running a current JDK version, and keep the JVM Arguments simple. There have been several years of bug fixes, memory leak fixes, threading improvements, and Garbage Collection enhancements, and therefore we recommend JDK 1.8.0_212 or newer.
Additionally, keeping your JVM Heap size below 16GB per our JVM Best Practices encourages horizontal scaling and prevents you from creating a monolithic JVM. Following this methodology will ensure you achieve a performant and stable instance of Jenkins, and will ultimately improve the experience for the end-users.
Please ensure you are using ONLY the following arguments unless directed by a Support Engineer, and if you have a question about adding an argument, please engage our Support Team.
If you are unsure how to add Java arguments to Jenkins, please see: How to add Java arguments to Jenkins.
Jenkins requires a restart for the JVM arguments to be picked up by the JVM.
JVM Heap Size
Heap Size should be determined by the current needs of the application. Per our JVM Best Practices we recommend a minimum of 2GB (4GB for Production Instances) and a maximum of 16GB (anything larger should be scaled horizontally.) It is important, as a JVM Administrator, to plan for heap to be increased as usage increases, and scale horizontally when the recommended maximum heap size of 16GB is reached. Monitoring Garbage Collection logging is the recommended way to predict how much heap you need.
JVM Recommended Arguments
Please implement the following arguments ensuring ${LOGDIR}
is replaced by a valid path on your operating system in which Jenkins has permission to write.
The arguments requiring a valid filesystem path (Xloggc , Xlog , -XX:HeapDumpPath , -XX:ErrorFile and -XX:LogFile ) should be adjusted for your OS (i.e. a slash in Linux becomes a backslash in Windows). Please consider using a path to local disk drives instead of network disk volumes for those arguments (especially Xlog / Xloggc ) in order to avoid I/O performance issues.
|
-
JDK 11
-XX:+AlwaysPreTouch -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${LOGDIR} -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+ParallelRefProcEnabled -XX:+DisableExplicitGC -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions -Xlog:gc*=info,gc+heap=debug,gc+ref*=debug,gc+ergo*=trace,gc+age*=trace:file=${LOGDIR}/gc.log:utctime,pid,level,tags:filecount=2,filesize=100M -XX:ErrorFile=${LOGDIR}/hs_err_%p.log -XX:+LogVMOutput -XX:LogFile=${LOGDIR}/jvm.log # Heap setting for CloudBees CI on modern cloud platforms: # -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0 # Heap setting for CloudBees CI on traditional platforms: # Heap Size `-Xmx` and `-Xms` should be set to the same value, and determined by following the above section "JVM Heap Size"
The parameters are explained in-depth in JDK 11 Oracle Documentation as well as our JVM Best Practice.
To increase GC logs to a longer period of time, we suggest increasing the value of the arguments
-Xlog
optionfilecount=2
and/orfilesize=100M
and as ultimate option usefile=${LOGDIR}/gc-%t.log
instead offile=${LOGDIR}/gc.log
. With the parameter%t
, the JVM create a new set of GC files each time that the instance is restarted. It is well known that when the GC log folder gets big enough in terms of size, the support bundle might produce performance issues in the instance given that it needs to compress all of them.Omitting -XX:+UnlockDiagnosticVMOptions
or-XX:+UnlockExperimentalVMOptions
might cause your instance to fail to startup. -
JDK 8
-XX:+AlwaysPreTouch -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${LOGDIR} -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+ParallelRefProcEnabled -XX:+DisableExplicitGC -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions -verbose:gc -Xloggc:${LOGDIR}/gc.log -XX:NumberOfGCLogFiles=2 -XX:+UseGCLogFileRotation -XX:GCLogFileSize=100m -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCCause -XX:+PrintTenuringDistribution -XX:+PrintReferenceGC -XX:+PrintAdaptiveSizePolicy -XX:ErrorFile=${LOGDIR}/hs_err_%p.log -XX:+LogVMOutput -XX:LogFile=${LOGDIR}/jvm.log # Heap setting for CloudBees CI on modern cloud platforms: # -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0 # Heap setting for CloudBees CI on traditional platforms: # Heap Size `-Xmx` and `-Xms` should be set to the same value, and determined by following the above section "JVM Heap Size"
The parameters are explained in-depth in JDK 8 Oracle Documentation as well as our JVM Best Practice.
To increase GC logs to a longer period of time, we suggest increasing the value of the arguments
-XX:GCLogFileSize
and-XX:NumberOfGCLogFiles
and as ultimate option use-Xloggc:${LOGDIR}/gc-%t.log
instead of-Xloggc:${LOGDIR}/gc.log
. With the parameter%t
, the JVM create a new set of GC files each time that the instance is restarted. It is well known that when the GC log folder gets big enough in terms of size, the support bundle might produce performance issues in the instance given that it needs to compress all of them.
B. Ulimit Settings [just for Linux OS]
ulimit -c
and ulimit -f
should be set to unlimited for the user that starts Jenkins. ulimit -c
set to unlimited will allow core files to be generated successfully. The core files include full thread dumps and core files generated by the JVM in the event of a JVM crash. ulimit -f
should be set to unlimited to ensure that files are not truncated during their generation.
ulimit -n
should be set to 4096 (soft) and 8192 (hard)
ulimit -u
should be set to 30654 (soft) and 30654 (hard)
The ulimit -n
and ulimit -u
setting recommendations are taken from: Memory problem: 'unable to create new native thread'
Please see this KB Article for setting ulimits in Docker: Too many open files.
To have the ulimits settings persist for the user that runs Jenkins, they will need to be added to /etc/security/limits.conf
.
Your /etc/security/limits.conf
file should like similar to the example below, for the user that starts jenkins in your environment:

C. Java Home Environment Variable
It is recommended to set the JAVA_HOME
environment variable in both Linux and Windows environments. The Java JDK’s bin directory should also be in the PATH
environment variable. This will allow for easier access to Java JDK commands, such as jstack
and jmap
. Please note that the jstack
command is only available in the Java JDK, not the Java JRE.
Please also note that core files and heapdump files can be large and are generated in the working directory of the JVM. Thus, it is recommended that there is adequate disk space for that directory.
For example, on Windows, the PATH
environment variable will include: C:\Program Files\Java\jdk1.8.0_202\bin
Running jstack in terminal window or command prompt should generate the following output:

D. Log Startup Timing Info
Per our Startup Troubleshooting Guide: it is recommended to set -Djenkins.model.Jenkins.logStartupPerformance=true
in your Jenkins environment to have Jenkins output additional data to the jenkins.log, showing how much time is being spent in a specific component during startup.
E. Jenkins Admin Should Be Able To Generate A Support Bundle via jenkins-cli.jar
This will allow Jenkins admins to generate support bundles in the event the Jenkins User Interface (UI) is not available, even though Jenkins is running.
Please note that Jenkins accounts must have the Overall/Read account permission to access the CLI.
The jenkins-cli.jar
can be downloaded from JENKINS_URL/cli.
More information about the jenkins-cli.jar
support command can be viewed by navigating to JENKINS_URL/cli/command/support .
Additional information about the jenkins-cli.jar
can be found at jenkins-cli.jar wiki page.
Please see the screenshot below showing a support bundle generated via jenkins-cli.jar
using the username and password.
The jenkins-cli.jar
authentication can be setup to use SSH keys if you prefer, as documented in the jenkins-cli.jar wiki page
linked above.
