Issue
-
Jenkins is getting slow and unresponsive intermittently
-
Blocked threads trying to get security warnings data. For example:
Handling GET /manage from 127.0.0.1 : qwerty-12345 Jenkins/manage.jelly "Handling GET /manage from 127.0.0.1 : qwerty-12345 Jenkins/manage.jelly" Id=389 Group=main BLOCKED on com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningDataProvider@caefc6b owned by "Beekeeper.analysis [#1]" Id=91 at com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningDataProvider.get(SecurityWarningDataProvider.java:70) - blocked on com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningDataProvider@caefc6b at com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningsMonitor.getSecurityWarningData(SecurityWarningsMonitor.java:36) at com.cloudbees.jenkins.plugins.assurance.model.Beekeeper.getSecurityWarnings(Beekeeper.java:250) at com.cloudbees.jenkins.plugins.assurance.SecurityWarningsWatch.isActivated(SecurityWarningsWatch.java:58) at jenkins.model.Jenkins.lambda$getActiveAdministrativeMonitors$0(Jenkins.java:2125)
-
Slows requests blocked by the Beekeeper security warning monitor. For example:
"Handling GET /manage from 127.0.0.1 : qwerty-12345 Jenkins/manage.jelly" Id=30658 BLOCKED on com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningDataProvider@a5c14ec owned by "Handling GET /manage from 127.0.0.1 : qwerty-12345 Jenkins/manage.jelly" Id=32013 at com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningDataProvider.get(SecurityWarningDataProvider.java:70) - blocked on com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningDataProvider@a5c14ec at com.cloudbees.jenkins.plugins.assurance.model.SecurityWarningsMonitor.getSecurityWarningData(SecurityWarningsMonitor.java:36) at com.cloudbees.jenkins.plugins.assurance.model.Beekeeper.getSecurityWarnings(Beekeeper.java:250) at com.cloudbees.jenkins.plugins.assurance.SecurityWarningsWatch.isActivated(SecurityWarningsWatch.java:58) at jenkins.model.Jenkins.lambda$getActiveAdministrativeMonitors$0(Jenkins.java:2140) at jenkins.model.Jenkins$$Lambda$254/848057038.test(Unknown Source) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) at java.util.Iterator.forEachRemaining(Iterator.java:116) ...
Environment
-
CloudBees Jenkins Enterprise - Managed controller (CJEMM) >= 2.107.x
-
CloudBees Jenkins Enterprise - Operations Center (CJEOC) >= 2.107.x
-
CloudBees Assurance Plugin >= 2.89.0.4
Related Issue(s)
-
FNDJEN-470 - Beekeeper plugin freezes Jenkins UI when the connection is slow against the security-warnings endpoint
Explanation
The release of CloudBees Jenkins Platform - Client controller 2.32.2.6 and the version 2.107.0.1 of the CloudBees Assurance Plugin introduced a performance issue in environments where the Beekeeper server is not reachable.
This version of the CloudBees Assurance Program plugin introduce a Security Warnings Monitor that detects new security upgrades information and displays a warning when one is available. The monitor retrieves information from https://beekeeper-server.cloudbees.com/api/security-warnings with a 60 seconds timeout. When this endpoint is not reachable and does not fail quickly, this causes thread contentions for 60 seconds. Many UI interactions are impacted. This can slow down a Jenkins instance and can lead to serious performance issues.
Resolution
The problem has been fixed in the version 2.138.0.5 of the CloudBees Assurance Plugin available since the release 2.164.2.1. The solution is to upgrade Jenkins to 2.164.2.1 or later.
Workaround
The workaround would be to disable the administrative monitor "Security Warnings Monitor" (com.cloudbees.jenkins.plugins.assurance.SecurityWarningsWatch
) in Manage Jenkins > Configure System > Administrative monitors configuration.