Memory leak caused by regression in C2 JIT Compiler in Java 11.0.16

3 minute readKnowledge base

Issue

  • After upgrading to Java 11.0.16, Jenkins restarts randomly due to memory issues. It either crashes or is killed by the OOM Killer.

Environment

Related Issue(s)

Explanation

Java 11.0.16 has a known regression issue JDK-8292260 in the C2 JIT Compiler that impacts memory consumption tremendously. It causes the JVM to crash. Though in many cases the OOM Killer would kill the instance before it crashes.

The impact depends on the environment and activity on the server. Based on feedbacks, it impacts rather active instances but also batch workloads (launch of multiple concurrent builds at once). Though it is rather unpredictable.

Resolution

CloudBees CI on Modern Platforms and CloudBees CI on Traditional Platforms container images

Starting from CloudBees CI on modern cloud platforms 2.361.1.2 the container images include Java 11.0.16.1.

The solution is to upgrade CloudBees CI to version 2.361.1.2 or later.

CloudBees CI on Traditional Platforms

The solution is to upgrade Java to version 11.0.16.1 or later.

Workaround

CloudBees CI on Traditional Platforms

A workaround is to downgrade Java to version 11.0.15.

CloudBees CI on Modern Platforms

You can use product versions which do not contain the impacted Java version (11.0.16), which are CloudBees CI on modern cloud platforms 2.346.2.2 and earlier, as well as CloudBees CI on modern cloud platforms 2.361.1.2 or later. Note that if you want to move to an earlier version of CloudBees CI, you first need to restore from your backup taken before the upgrade, and then upgrade again to your new desired release version, downgrades are not supported.

Alternatively, if you want to use one of the impacted product versions, and are encountering this issue, you can build custom container images with Java version 11.0.16.1 or later.

Here is an example of how to build a custom controller image for version 2.346.4.1 using Temurin (https://adoptium.net/) 11.0.16.1:

#/bin/sh
wget https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.16.1%2B1/OpenJDK11U-jdk_x64_linux_hotspot_11.0.16.1_1.tar.gz
tar -xf OpenJDK11U-jdk_x64_linux_hotspot_11.0.16.1_1.tar.gz
docker build -t cloudbees-core-mm:2.346.4.1-jdk11.0.16.1 .
▼
FROM cloudbees/cloudbees-core-mm:2.346.4.1
USER root
RUN /bin/sh -c dnf erase java-11-openjdk java-11-openjdk-devel && rm -rf /usr/lib/jvm/java-11-openjdk-11.0.16.0.8-1.el8_6.x86_64
COPY jdk-11.0.16.1+1 /usr/lib/jvm/jdk-11.0.16.1+1
RUN alternatives --install /usr/bin/java java /usr/lib/jvm/jdk-11.0.16.1+1/bin/java 99
RUN alternatives --install /usr/bin/jstack jstack /usr/lib/jvm/jdk-11.0.16.1+1/bin/jstack 99
RUN alternatives --install /usr/bin/jps jps /usr/lib/jvm/jdk-11.0.16.1+1/bin/jps 99
RUN alternatives --install /usr/bin/jcmd jcmd /usr/lib/jvm/jdk-11.0.16.1+1/bin/jcmd 99
USER 1000
▼

You’ll have to adjust the image tag when running docker build to one you can push to your private container registry.

Also, note that this example is for the managed controller, for your operations center you would change the Dockerfile line from:

FROM cloudbees/cloudbees-core-mm:2.346.4.1

to

FROM cloudbees/cloudbees-cloud-core-oc:2.346.4.1

As well as change the docker tag when running docker build.

Once you push these images to your container registry, to use them:

  1. For your Operations center, override the helm values OperationsCenter.Image.dockerImage to be the container registry location for your custom image, and OperationsCenter.ImagePullSecrets to be the name of the Kubernetes secret to use to pull images from your container registry.

  2. For your managed controllers, follow Using private image repository in Operations Center controller provisioning.