Controller is OOMKilled due to native memory growth not visible in Native Memory Tracking

Last Reviewed:2026-10-15()
3 minute readKnowledge base

Issue

A controller running in a Linux container is killed by the kernel or container runtime’s out-of-memory (OOM) mechanism, even though Java heap usage remains well below the container memory limit at the time of the restart. Native Memory Tracking (NMT) and other JVM-level memory metrics (heap, metaspace, thread stacks, code cache, GC structures) also remain below the container memory limit, leaving a large, growing gap between the container’s actual memory usage and the memory that JVM-level tooling accounts for.

Environment

Explanation

Native Memory Tracking only accounts for memory that the JVM itself requests through its own malloc/mmap calls, categorized into areas such as Java Heap, Class (metaspace), Thread, Code, and GC.

On Linux, the glibc C library’s own malloc implementation can create multiple independent memory arenas to reduce allocation contention between threads, with the number of arenas scaling with the number of CPU cores. By default, it will create 'number of cores times 8' memory arenas. As many threads concurrently allocate and free native memory, for example from plugins or libraries that allocate memory outside the JVM’s own tracked subsystems, glibc can retain freed memory inside these per-thread arenas as fragmented free lists instead of returning it to the operating system. This growth happens inside glibc’s own allocator bookkeeping, which is outside of what NMT tracks, so it does not appear in an NMT summary even though it counts toward the container’s memory limit.

Identify whether this is the cause

  1. Collect an NMT summary by following Diagnosing Java Native Memory Issues, and confirm that Java Heap, Class (metaspace), Thread, Code, GC, and the other NMT-tracked categories together remain small relative to the container memory limit.

  2. Compare the container’s actual memory usage (for example, from kubectl top pod, the cgroup memory.current or memory.usage_in_bytes file, or your monitoring tooling) against the sum of the NMT-tracked categories from the previous step.

  3. If there is a large discrepancy between the container’s memory usage and the NMT-tracked categories, this is consistent with glibc malloc arena growth rather than a JVM heap or metaspace leak.

Resolution

Setting the MALLOC_ARENA_MAX environment variable to a lower number limits the number of glibc malloc arenas the controller process can create, which causes malloc to free unused memory back to the OS more often. This works because malloc will only trim arenas when they are accessed. Having less arenas means they are accessed more often.

MALLOC_ARENA_MAX is a process environment variable read by glibc at process startup. It is not a JVM argument, so it cannot be set through Java Options or System Properties.

  • For CloudBees CI on modern platforms, Update the Advanced configuration YAML field with the following (If using HA active/active use Deployment instead of StatefulSet)

apiVersion: "apps/v1"
kind: "StatefulSet"
spec:
  template:
    spec:
      containers:
      - env:
        - name: "MALLOC_ARENA_MAX"
          value: "4"
▼

  • For CloudBees CI on traditional platforms, add MALLOC_ARENA_MAX=4 to the environment used to start the Jenkins service (for example, the service’s environment file or init script).

  • Restart the controller for the change to take effect.

References

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.