CloudBees CI running in a container with Linux kernel 6.12.0 and newer getting OOMKilled

Issue

When running CloudBees CI in a container with Linux kernel 6.12.0 and newer, the container may get OOMKilled. This issue is related to a change in Linux Kernel 6.12 and newer versions related to cgroups, tracked by https://bugs.openjdk.org/browse/JDK-8349988 and https://bugs.openjdk.org/browse/JDK-8347811. As a result of this change, instead of the container getting 1/4 of the container memory limit by default, it gets 1/4 of the host memory limit. Using the recommended -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0 as JVM options will still result in the container getting 50% of the host memory limit (instead of the container memory limit), which is likely to cause the container to get OOMKilled.

Environment

Resolution

Wait until CloudBees CI is running on Supported JDK version that has the fix for https://bugs.openjdk.org/browse/JDK-8349988, which is available in JDK 21 starting with update 10.

Workaround

If running CloudBees CI in a container with Linux kernel 6.12.0 or newer without a corrected JDK version, you can add -XX:MaxRAM with your current container memory limit to the Java Options for controllers and to OperationsCenter.JavaOpts helm values for CloudBees CI on modern cloud platforms operations center to ensure that the container gets the chosen memory limit instead of the host memory limit:

-XX:MaxRAM=Xg -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0
# For example, if the container memory limit is 4g, the JVM heap will get 2g if you use:
# -XX:MaxRAM=4g -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0
# For example, if the container memory limit is 3072m, the JVM heap will get 1536m if you use:
# -XX:MaxRAM=3072m -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0▼

If running CloudBees CI version 2.528.1.29783 through 2.541.1.35570, you can set -XX:MaxRAM for all controllers in a Kubernetes endpoint with one change, under the operations center Manage Jenkins Configure Controller Provisioning by checking the Set -XX:MaxRAM box. Subsequent CloudBees CI releases remove this option since they bundle a Java version with the fix.

Tested product/plugin versions

CloudBees CI 2.516.1.28665 through 2.541.1.35570

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.