Issue
-
I am running the Datadog Tracing Agent and my Jenkins JVM heap utilization is unexpectedly climbing over time.
Environment
-
CloudBees Jenkins Enterprise (CJE)
-
CloudBees Jenkins Enterprise - Managed controller (CJE-MM)
-
CloudBees Jenkins Enterprise - Operations center (CJE-OC)
Resolution
This issue has since been resolved by Datadog as noted in this GitHub Commit
The fixed version of 0.20 was released December 13, 2018 per the Release Notes. It is recommended to upgrade to the latest version of the agent to ensure you are taking advantage of improvements and new features.
Explanation
Upon initial observation of a client provided heap dump, a 5GB+ chunk of memory was observed as a potential leak suspect.
Further investigation proved that the Datadog Java Agent was leaking memory from the WeakConcurrentMap to the tune of 5GB, causing an OOM event which ultimately brought down Jenkins.
One instance of "com.blogspot.mydailyjava.weaklockfree.WeakConcurrentMap" loaded by "datadog.trace.bootstrap.DatadogClassLoader @ 0x1c0667058" occupies 5,450,446,944 (22.28%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class loader>"
The Datadog Tracing Agent is invoked through JVM Arguments: -javaagent:/etc/datadog-agent/dd-java-agent.jar