Memory Leak in Datadog Agent

Article ID:360028520471
1 minute readKnowledge base

Issue

  • I am running the Datadog Tracing Agent and my Jenkins JVM heap utilization is unexpectedly climbing over time.

indicative_of_memory_leak.png

Resolution

This issue has since been resolved by Datadog as noted in this GitHub Commit

The fixed version of 0.20 was released December 13, 2018 per the Release Notes. It is recommended to upgrade to the latest version of the agent to ensure you are taking advantage of improvements and new features.

Explanation

datadog_1.png

Upon initial observation of a client provided heap dump, a 5GB+ chunk of memory was observed as a potential leak suspect.

datadog_2.png

Further investigation proved that the Datadog Java Agent was leaking memory from the WeakConcurrentMap to the tune of 5GB, causing an OOM event which ultimately brought down Jenkins.

One instance of "com.blogspot.mydailyjava.weaklockfree.WeakConcurrentMap" loaded by "datadog.trace.bootstrap.DatadogClassLoader @ 0x1c0667058" occupies 5,450,446,944 (22.28%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class loader>"

The Datadog Tracing Agent is invoked through JVM Arguments: -javaagent:/etc/datadog-agent/dd-java-agent.jar