Prepare CloudBees CI for Support

Issue

What are the best practices to configure CloudBees CI from the support team?

Environment

Resolution

The following settings are known Best Practices, and should be used in your CloudBees CI environment to allow for easier support in the event of an issue.

A. Java Configuration

JVM Administration and Tuning is necessary for any Enterprise Java Application Deployment. As CloudBees CI is an Enterprise Java Application, the following arguments are considered to be best practices, based on historical data analysis of Garbage Collection Logs, Thread Dump Analysis, and Heap Dump Analysis.

It is important to ensure you are running an up-to-date JDK version (if you are using container images, this is already being done during the image build), and keep the JVM Arguments simple. There have been several years of bug fixes, memory leak fixes, threading improvements, and Garbage Collection enhancements, and therefore the latest available fix pack for the JVM is recommended. Ensure you are running a supported JVM by consulting:

Additionally, keeping your JVM Heap size below 16GB per our CloudBees Jenkins JVM troubleshooting encourages horizontal scaling and prevents you from creating a monolithic JVM. Following this methodology will ensure you achieve a performant and stable instance of CloudBees CI, and will ultimately improve the experience for the end-users.

Please ensure you are using ONLY the following arguments unless directed by a Support Engineer, and if you have a question about adding an argument, please engage our support team.

If you are unsure how to add Java arguments to CloudBees CI, please see: How to add Java arguments to Jenkins?

CloudBees CI requires a restart for the JVM arguments to be picked up.

JVM Heap Size

Heap Size should be determined by the current needs of the application. Per our JVM Best Practices we recommend a minimum of 2GB (4GB for Production Instances) and a maximum of 16GB (anything larger should be scaled horizontally). It is important, as a JVM Administrator, to plan for heap to be increased as usage increases, and scale horizontally when the recommended maximum heap size of 16GB is reached. Monitoring Garbage Collection logging is the recommended way to predict how much heap you need.

You can scale controllers horizontally by:

JVM Recommended Arguments

Please implement the following arguments ensuring ${LOGDIR} is replaced by a valid path on your operating system in which CloudBees CI has permission to write.

On CloudBees CI on traditional platforms this path should generally be on disk that is local to the controller, not a shared filesystem. A suggested path on Linux would be under /var/log/cloudbees-core-cm/ for a controller, and /var/log/cloudbees-core-oc/ for an operations center, which should usually already exist. When choosing an alternate directory for logs that does not exist, it will need to be created by you and have the correct permissions and ownership set. Here’s a basic example of how to create the directory for a controller:

sudo mkdir -p /var/log/cloudbees-core-cm/
sudo chmod 700 /var/log/cloudbees-core-cm/
sudo chown -R cloudbees-core-cm:cloudbees-core-cm /var/log/cloudbees-core-cm/▼

Here’s a basic example of how to create the directory for an operations center:

sudo mkdir -p /var/log/cloudbees-core-oc/
sudo chmod 700 /var/log/cloudbees-core-oc/
sudo chown -R cloudbees-core-oc:cloudbees-core-oc /var/log/cloudbees-core-oc/▼

On CloudBees CI on modern cloud platforms this path could be on any Persistent Volume mounted to your pod, but consider if you want to omit thie directory chosen from your backups when you are backing up Persistent Volumes in your cluster.

The arguments requiring a valid filesystem path (Xloggc, Xlog, -XX:HeapDumpPath, -XX:ErrorFile and -XX:LogFile) should be adjusted for your OS (i.e. a slash in Linux becomes a backslash in Windows). Please consider using a path to local disk drives instead of network disk volumes for those arguments (especially Xlog / Xloggc) in order to avoid I/O performance issues.

JDK 11 and 17

-XX:+AlwaysPreTouch
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=${LOGDIR}
-XX:+UseG1GC
-XX:+UseStringDeduplication
-XX:+ParallelRefProcEnabled
-XX:+DisableExplicitGC
-XX:+UnlockDiagnosticVMOptions
-XX:+UnlockExperimentalVMOptions
-Xlog:gc*=info,gc+heap=debug,gc+ref*=debug,gc+ergo*=trace,gc+age*=trace:file=${LOGDIR}/gc.log:utctime,pid,level,tags:filecount=2,filesize=100M
-XX:ErrorFile=${LOGDIR}/hs_err_%p.log
-XX:+LogVMOutput
-XX:LogFile=${LOGDIR}/jvm.log
# Heap setting for CloudBees CI on modern cloud platforms:
#   -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0
# Heap setting for CloudBees CI on traditional platforms:
#   Heap Size `-Xmx` and `-Xms` should be set to the same value, and determined by following the above section "JVM Heap Size"

The parameters are explained in-depth in JDK 11 Oracle Documentation as well as our JVM Best Practice.

To increase GC logs to a longer period of time, we suggest increasing the value of the arguments -Xlog option filecount=2 and/or filesize=100M and as ultimate option use file=${LOGDIR}/gc-%t.log instead of file=${LOGDIR}/gc.log. With the parameter %t, the JVM create a new set of GC files each time that the instance is restarted. It is well known that when the GC log folder gets big enough in terms of size, the support bundle might produce performance issues in the instance given that it needs to compress all of them.

Omitting -XX:+UnlockDiagnosticVMOptions or -XX:+UnlockExperimentalVMOptions might cause your instance to fail to startup.

JDK 8

-XX:+AlwaysPreTouch
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=${LOGDIR}
-XX:+UseG1GC
-XX:+UseStringDeduplication
-XX:+ParallelRefProcEnabled
-XX:+DisableExplicitGC
-XX:+UnlockDiagnosticVMOptions
-XX:+UnlockExperimentalVMOptions
-verbose:gc
-Xloggc:${LOGDIR}/gc.log
-XX:NumberOfGCLogFiles=2
-XX:+UseGCLogFileRotation
-XX:GCLogFileSize=100m
-XX:+PrintGC
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintHeapAtGC
-XX:+PrintGCCause
-XX:+PrintTenuringDistribution
-XX:+PrintReferenceGC
-XX:+PrintAdaptiveSizePolicy
-XX:ErrorFile=${LOGDIR}/hs_err_%p.log
-XX:+LogVMOutput
-XX:LogFile=${LOGDIR}/jvm.log
# Heap setting for CloudBees CI on modern cloud platforms:
#   -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0
# Heap setting for CloudBees CI on traditional platforms:
#   Heap Size `-Xmx` and `-Xms` should be set to the same value, and determined by following the above section "JVM Heap Size"

The parameters are explained in-depth in JDK 8 Oracle Documentation as well as our JVM Best Practice.

To increase GC logs to a longer period of time, we suggest increasing the value of the arguments -XX:GCLogFileSize and -XX:NumberOfGCLogFiles and as ultimate option use -Xloggc:${LOGDIR}/gc-%t.log instead of -Xloggc:${LOGDIR}/gc.log. With the parameter %t, the JVM create a new set of GC files each time that the instance is restarted. It is well known that when the GC log folder gets big enough in terms of size, the support bundle might produce performance issues in the instance given that it needs to compress all of them.

B. Ulimit Settings [just for Linux OS]

Please be aware that if CloudBees CI on traditional platforms runs under control of systemd, then defining the ulimits as shown below within /etc/security/limits.conf has no effect on CloudBees CI. Parameters like LimitNoFile and LimitNoProc must be set within the systemd service unit file.

ulimit -c and ulimit -f should be set to unlimited for the user that starts CloudBees CI. ulimit -c set to unlimited will allow core files to be generated successfully. The core files include full thread dumps and core files generated by the JVM in the event of a JVM crash. ulimit -f should be set to unlimited to ensure that files are not truncated during their generation.

ulimit -n should be set to 4096 (soft) and 8192 (hard)

ulimit -u should be set to 30654 (soft) and 30654 (hard)

The ulimit -n and ulimit -u setting recommendations are taken from: Memory problem: 'unable to create new native thread'

Please see this KB Article for setting ulimits in Docker: Too many open files.

To have the ulimits settings persist for the user that runs CloudBees CI, they will need to be added to /etc/security/limits.conf.

Your /etc/security/limits.conf file should like similar to the example below, for the user that starts CloudBees CI in your environment:

C. Java Home Environment Variable

It is recommended to set the JAVA_HOME environment variable in both Linux and Windows environments. The Java JDK’s bin directory should also be in the PATH environment variable. This will allow for easier access to Java JDK commands, such as jstack and jmap. Please note that the jstack command is only available in the Java JDK, not the Java JRE.

Please also note that core files and heapdump files can be large and are generated in the working directory of the JVM. Thus, it is recommended that there is adequate disk space for that directory.

For example, on Windows, the PATH environment variable will include: C:\Program Files\Java\jdk1.8.0_202\bin or similar for other Java versions

Running jstack in terminal window or command prompt should generate the following output:

D. Log Startup Timing Info

Per our Startup Troubleshooting Guide: it is recommended to set -Djenkins.model.Jenkins.logStartupPerformance=true in your CloudBees CI environment to have CloudBees CI output additional data to the log, showing how much time is being spent in a specific component during startup.

E. CloudBees CI Admin Should Be Able To Generate A Support Bundle via jenkins-cli.jar

This will allow CloudBees CI admins to generate support bundles in the event the CloudBees CI User Interface (UI) is not available, even though CloudBees CI is running.

Please note that CloudBees CI accounts must have the Overall/Read account permission to access the CLI.

The jenkins-cli.jar can be downloaded from CONTROLLER_URL/cli.

More information about the jenkins-cli.jar support command can be viewed by navigating to CONTROLLER_URL/cli/command/support .

Additional information about the jenkins-cli.jar can be found at Jenkins CLI.

Please see the screenshot below showing a support bundle generated via jenkins-cli.jar using the username and password. The jenkins-cli.jar authentication can be setup to use SSH keys if you prefer, as documented in the jenkins-cli.jar wiki page linked above.