CloudBees Usage Analyzer Plugin causing heap memory pressure on monolith controllers

2 minute readKnowledge base

Issue

When a controller has the CloudBees Usage Analyzer Plugin plugin version 2.x, as well as a large number of jobs, for example more than 5,000 jobs on one controller (each of those jobs could have multiple branches if they are multibranch pipelines, as well as build history for each job), the instance can slow down and become unresponsive, and you will be unable to access the controllers web UI.

When reviewing Garbage Collection logs of the controller, you will see the typical pattern showing Consecutive Full GC: https://blog.ycrash.io/2021/10/15/interesting-garbage-collection-patterns/

When reviewing a heap dump (How to generate a heap dump?), you will notice that greater than 90% of heap memory is consumed by Plugin Usages Analyzer thread:

For example in Eclipse MAT, in the "Leak suspects" report you may see a leak suspect with:

71,566 instances of "org.jenkinsci.plugins.workflow.job.WorkflowRun", loaded by "jenkins.util.AntClassLoader @ …​" occupy 34,039,999,568 (92.73%) bytes.

And under the "Common Path to the Accumulation Point" you will see:

java.lanq.Thread @ Plugin Usages Analyzer thread Thread

Resolution

We recommend upgrading to CloudBees CI 2.440.3.7 or later, which contains:

Fixed excessive memory usage by the Plugin Usage Analyzer plugin

The Plugin Usage Analyzer had a flaw where it held on to all of the Pipeline runs that it wanted to analyze before performing the analysis of their usage. This caused CloudBees CI to use an excessive amount of memory when the analysis was running. This is now fixed by the Plugin Usage Analyzer plugin only holding in memory the pipeline runs of the current pipeline that is being analyzed (a maximum of 5).

Workaround

When the plugin is disabled and you want to locate jobs that are using a specific plugin, for example the maven-plugin, you can run the following command which will list the job config.xml files for the jobs that use that plugin (note that this does not work for Pipeline steps, only plugins that save configuration settings to the jobs config.xml):

find jenkins-home/jobs -name config.xml -exec grep -l 'plugin="maven-plugin' {} \; jenkins-home/jobs/my-maven-job/config.xml

Tested product/plugin versions

  • CloudBees Jenkins Platform - controller version 2.346.4.1

  • CloudBees Usage Analyzer Plugin plugin version 2.15

  • Controllers which have been impacted by this issue have had from ~4000 up to 200,000 jobs on them.