Prepare Jenkins for Support

Issue

Best Practices to Support Jenkins

Environment

These best practices are for the following products and should not be deployed for CI-Modern

Note

Please be aware that if Jenkins runs under control of systemd, then defining the ulimits as shown below within /etc/security/limits.conf has no effect on Jenkins. Parameters like LimitNoFile and LimitNoProc must be set within the systemd service unit file.

Resolution

The following settings are known Best Practices, and should be used in your Jenkins environment to allow for easier support in the event of an issue.

A. Java Configuration

CloudBees Supported Java Version

JVM Administration and Tuning is necessary for any Enterprise Java Application Deployment. As Jenkins is an Enterprise Java Application, the following arguments are considered to be best practices, based on historical data analysis of Garbage Collection Logs, Thread Dump Analysis, and Heap Dump Analysis.

In the last several years with Java 1.8.0, fine-tuning Garbage Collection settings was considered a common practice, but ultimately masks underlying problems within the JDK. Because of this, it was also common practice to allocate more RAM and CPU resources to the JVM to keep it running smoothly. While this will help in the short term, it still masks underlying problems. These methodologies are described in-depth in this article.

In today’s methodologies, it is more important to ensure you are running a current JDK version, and keep the JVM Arguments simple. There have been several years of bug fixes, memory leak fixes, threading improvements, and Garbage Collection enhancements, and therefore we recommend JDK 1.8.0_212 or newer.

Additionally, keeping your JVM Heap size below 16GB per our JVM Best Practices encourages horizontal scaling and prevents you from creating a monolithic JVM. Following this methodology will ensure you achieve a performant and stable instance of Jenkins, and will ultimately improve the experience for the end-users.

Please ensure you are using ONLY the following arguments unless directed by a Support Engineer, and if you have a question about adding an argument, please engage our Support Team.

If you are unsure how to add Java arguments to Jenkins, please see: How to add Java arguments to Jenkins.

Jenkins requires a restart for the JVM arguments to be picked up by the JVM.

JVM Heap Size

Heap Size should be determined by the current needs of the application. Per our JVM Best Practices we recommend a minimum of 2GB (4GB for Production Instances) and a maximum of 16GB (anything larger should be scaled horizontally.) It is important, as a JVM Administrator, to plan for heap to be increased as usage increases, and scale horizontally when the recommended maximum heap size of 16GB is reached. Monitoring Garbage Collection logging is the recommended way to predict how much heap you need.

JVM Recommended Arguments

Please implement the following arguments ensuring ${LOGDIR} is replaced by a valid path on your operating system in which Jenkins has permission to write.

The arguments requiring a valid filesystem path (Xloggc, Xlog, -XX:HeapDumpPath, -XX:ErrorFile and -XX:LogFile) should be adjusted for your OS (i.e. a slash in Linux becomes a backslash in Windows). Please consider using a path to local disk drives instead of network disk volumes for those arguments (especially Xlog / Xloggc) in order to avoid I/O performance issues.

JDK 11

  -XX:+AlwaysPreTouch
  -XX:+HeapDumpOnOutOfMemoryError
  -XX:HeapDumpPath=${LOGDIR}
  -XX:+UseG1GC
  -XX:+UseStringDeduplication
  -XX:+ParallelRefProcEnabled
  -XX:+DisableExplicitGC
  -XX:+UnlockDiagnosticVMOptions
  -XX:+UnlockExperimentalVMOptions
  -Xlog:gc*=info,gc+heap=debug,gc+ref*=debug,gc+ergo*=trace,gc+age*=trace:file=${LOGDIR}/gc.log:utctime,pid,level,tags:filecount=2,filesize=100M
  -XX:ErrorFile=${LOGDIR}/hs_err_%p.log
  -XX:+LogVMOutput
  -XX:LogFile=${LOGDIR}/jvm.log
  # Heap setting for CloudBees CI on modern cloud platforms:
  #   -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0
  # Heap setting for CloudBees CI on traditional platforms:
  #   Heap Size `-Xmx` and `-Xms` should be set to the same value, and determined by following the above section "JVM Heap Size"

The parameters are explained in-depth in JDK 11 Oracle Documentation as well as our JVM Best Practice.

To increase GC logs to a longer period of time, we suggest increasing the value of the arguments -Xlog option filecount=2 and/or filesize=100M and as ultimate option use file=${LOGDIR}/gc-%t.log instead of file=${LOGDIR}/gc.log. With the parameter %t, the JVM create a new set of GC files each time that the instance is restarted. It is well known that when the GC log folder gets big enough in terms of size, the support bundle might produce performance issues in the instance given that it needs to compress all of them.

Omitting -XX:+UnlockDiagnosticVMOptions or -XX:+UnlockExperimentalVMOptions might cause your instance to fail to startup.

JDK 8

  -XX:+AlwaysPreTouch
  -XX:+HeapDumpOnOutOfMemoryError
  -XX:HeapDumpPath=${LOGDIR}
  -XX:+UseG1GC
  -XX:+UseStringDeduplication
  -XX:+ParallelRefProcEnabled
  -XX:+DisableExplicitGC
  -XX:+UnlockDiagnosticVMOptions
  -XX:+UnlockExperimentalVMOptions
  -verbose:gc
  -Xloggc:${LOGDIR}/gc.log
  -XX:NumberOfGCLogFiles=2
  -XX:+UseGCLogFileRotation
  -XX:GCLogFileSize=100m
  -XX:+PrintGC
  -XX:+PrintGCDateStamps
  -XX:+PrintGCDetails
  -XX:+PrintHeapAtGC
  -XX:+PrintGCCause
  -XX:+PrintTenuringDistribution
  -XX:+PrintReferenceGC
  -XX:+PrintAdaptiveSizePolicy
  -XX:ErrorFile=${LOGDIR}/hs_err_%p.log
  -XX:+LogVMOutput
  -XX:LogFile=${LOGDIR}/jvm.log
  # Heap setting for CloudBees CI on modern cloud platforms:
  #   -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0
  # Heap setting for CloudBees CI on traditional platforms:
  #   Heap Size `-Xmx` and `-Xms` should be set to the same value, and determined by following the above section "JVM Heap Size"

The parameters are explained in-depth in JDK 8 Oracle Documentation as well as our JVM Best Practice.

To increase GC logs to a longer period of time, we suggest increasing the value of the arguments -XX:GCLogFileSize and -XX:NumberOfGCLogFiles and as ultimate option use -Xloggc:${LOGDIR}/gc-%t.log instead of -Xloggc:${LOGDIR}/gc.log. With the parameter %t, the JVM create a new set of GC files each time that the instance is restarted. It is well known that when the GC log folder gets big enough in terms of size, the support bundle might produce performance issues in the instance given that it needs to compress all of them.

B. Ulimit Settings [just for Linux OS]

ulimit -c and ulimit -f should be set to unlimited for the user that starts Jenkins. ulimit -c set to unlimited will allow core files to be generated successfully. The core files include full thread dumps and core files generated by the JVM in the event of a JVM crash. ulimit -f should be set to unlimited to ensure that files are not truncated during their generation.

ulimit -n should be set to 4096 (soft) and 8192 (hard)

ulimit -u should be set to 30654 (soft) and 30654 (hard)

The ulimit -n and ulimit -u setting recommendations are taken from: Memory problem: 'unable to create new native thread'

Please see this KB Article for setting ulimits in Docker: Too many open files.

To have the ulimits settings persist for the user that runs Jenkins, they will need to be added to /etc/security/limits.conf.

Your /etc/security/limits.conf file should like similar to the example below, for the user that starts jenkins in your environment:

C. Java Home Environment Variable

It is recommended to set the JAVA_HOME environment variable in both Linux and Windows environments. The Java JDK’s bin directory should also be in the PATH environment variable. This will allow for easier access to Java JDK commands, such as jstack and jmap. Please note that the jstack command is only available in the Java JDK, not the Java JRE.

Please also note that core files and heapdump files can be large and are generated in the working directory of the JVM. Thus, it is recommended that there is adequate disk space for that directory.

For example, on Windows, the PATH environment variable will include: C:\Program Files\Java\jdk1.8.0_202\bin

Running jstack in terminal window or command prompt should generate the following output:

D. Log Startup Timing Info

Per our Startup Troubleshooting Guide: it is recommended to set -Djenkins.model.Jenkins.logStartupPerformance=true in your Jenkins environment to have Jenkins output additional data to the jenkins.log, showing how much time is being spent in a specific component during startup.

E. Jenkins Admin Should Be Able To Generate A Support Bundle via jenkins-cli.jar

This will allow Jenkins admins to generate support bundles in the event the Jenkins User Interface (UI) is not available, even though Jenkins is running.

Please note that Jenkins accounts must have the Overall/Read account permission to access the CLI.

The jenkins-cli.jar can be downloaded from JENKINS_URL/cli.

More information about the jenkins-cli.jar support command can be viewed by navigating to JENKINS_URL/cli/command/support .

Additional information about the jenkins-cli.jar can be found at jenkins-cli.jar wiki page.

Please see the screenshot below showing a support bundle generated via jenkins-cli.jar using the username and password. The jenkins-cli.jar authentication can be setup to use SSH keys if you prefer, as documented in the jenkins-cli.jar wiki page linked above.