Performance Issues caused by the Beekeeper Security Warnings Monitor

Article ID:360024928611
2 minute readKnowledge base

Issue

  • Jenkins is getting slow and unresponsive intermittently

  • Blocked threads trying to get security warnings data. For example:

Handling GET /manage from 127.0.0.1 : qwerty-12345 Jenkins/manage.jelly
"Handling GET /manage from 127.0.0.1 : qwerty-12345 Jenkins/manage.jelly" Id=389 Group=main BLOCKED on com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningDataProvider@caefc6b owned by "Beekeeper.analysis [#1]" Id=91
	at com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningDataProvider.get(SecurityWarningDataProvider.java:70)
	-  blocked on com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningDataProvider@caefc6b
	at com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningsMonitor.getSecurityWarningData(SecurityWarningsMonitor.java:36)
	at com.cloudbees.jenkins.plugins.assurance.model.Beekeeper.getSecurityWarnings(Beekeeper.java:250)
	at com.cloudbees.jenkins.plugins.assurance.SecurityWarningsWatch.isActivated(SecurityWarningsWatch.java:58)
	at jenkins.model.Jenkins.lambda$getActiveAdministrativeMonitors$0(Jenkins.java:2125)

  • Slows requests blocked by the Beekeeper security warning monitor. For example:

"Handling GET /manage from 127.0.0.1 : qwerty-12345 Jenkins/manage.jelly" Id=30658 BLOCKED on com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningDataProvider@a5c14ec owned by "Handling GET /manage from 127.0.0.1 : qwerty-12345 Jenkins/manage.jelly" Id=32013
	at com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningDataProvider.get(SecurityWarningDataProvider.java:70)
	-  blocked on com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningDataProvider@a5c14ec
	at com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningsMonitor.getSecurityWarningData(SecurityWarningsMonitor.java:36)
	at com.cloudbees.jenkins.plugins.assurance.model.Beekeeper.getSecurityWarnings(Beekeeper.java:250)
	at com.cloudbees.jenkins.plugins.assurance.SecurityWarningsWatch.isActivated(SecurityWarningsWatch.java:58)
	at jenkins.model.Jenkins.lambda$getActiveAdministrativeMonitors$0(Jenkins.java:2140)
	at jenkins.model.Jenkins$$Lambda$254/848057038.test(Unknown Source)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
	at java.util.Iterator.forEachRemaining(Iterator.java:116)
	...

Environment

Related Issue(s)

  • FNDJEN-470 - Beekeeper plugin freezes Jenkins UI when the connection is slow against the security-warnings endpoint

Explanation

The release of CloudBees Jenkins Platform - Client controller 2.32.2.6 and the version 2.107.0.1 of the CloudBees Assurance Plugin introduced a performance issue in environments where the Beekeeper server is not reachable.

This version of the CloudBees Assurance Program plugin introduce a Security Warnings Monitor that detects new security upgrades information and displays a warning when one is available. The monitor retrieves information from https://beekeeper-server.cloudbees.com/api/security-warnings with a 60 seconds timeout. When this endpoint is not reachable and does not fail quickly, this causes thread contentions for 60 seconds. Many UI interactions are impacted. This can slow down a Jenkins instance and can lead to serious performance issues.

Resolution

The problem has been fixed in the version 2.138.0.5 of the CloudBees Assurance Plugin available since the release 2.164.2.1. The solution is to upgrade Jenkins to 2.164.2.1 or later.

Workaround

The workaround would be to disable the administrative monitor "Security Warnings Monitor" (com.cloudbees.jenkins.plugins.assurance.SecurityWarningsWatch) in Manage Jenkins > Configure System > Administrative monitors configuration.