Issue
-
Jenkins is getting slow
-
Many blocked threads trying to compute the Beekeeper report. For example:
"Handling GET / from 127.0.0.1 : RequestHandlerThread[#xxx] View/index.jelly AdministrativeMonitorsDecorator/footer.jelly" [...] waiting for monitor entry [...]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.cloudbees.jenkins.plugins.assurance.Cached.get(Cached.java:37)
- waiting to lock <0x00000006445b5c28> (a com.cloudbees.jenkins.plugins.assurance.CloudBeesAssurance$2)
at com.cloudbees.jenkins.plugins.assurance.CloudBeesAssurance.getReport(CloudBeesAssurance.java:327)
at com.cloudbees.jenkins.plugins.assurance.CAPUpdateCenterFilter.getItem(CAPUpdateCenterFilter.java:51)
at com.cloudbees.jenkins.plugins.assurance.CAPUpdateCenterFilter.getCoreSource(CAPUpdateCenterFilter.java:110)
at com.cloudbees.jenkins.cjp.installmanager.CJPUpdateCenter.getCoreSource(CJPUpdateCenter.java:72)
at hudson.model.UpdateCenter$CoreUpdateMonitor.getData(UpdateCenter.java:1002)
at hudson.model.UpdateCenter$CoreUpdateMonitor.isActivated(UpdateCenter.java:997)
at jenkins.management.AdministrativeMonitorsDecorator.getActiveAdministrativeMonitors(AdministrativeMonitorsDecorator.java:91)
at jenkins.management.AdministrativeMonitorsDecorator.getActiveAdministrativeMonitorsCount(AdministrativeMonitorsDecorator.java:76)
at jenkins.management.AdministrativeMonitorsDecorator.shouldDisplay(AdministrativeMonitorsDecorator.java:142)
...
-
Many slows requests blocked by the Beekeeper report computation. For example:
"Handling GET /job/myjob/job/master/ from 127.0.0.1 : RequestHandlerThread[#xxxx] Job/index.jelly AdministrativeMonitorsDecorator/footer.jelly" Id=xxxxxx BLOCKED on com.cloudbees.jenkins.plugins.assurance.CloudBeesAssurance$2@xxxxxx owned by "Handling GET / from 127.0.0.1 : RequestHandlerThread[#xxx] View/index.jelly AdministrativeMonitorsDecorator/footer.jelly" Id=xxxxxx
at com.cloudbees.jenkins.plugins.assurance.Cached.get(Cached.java:37)
- blocked on com.cloudbees.jenkins.plugins.assurance.CloudBeesAssurance$2@7db3c53c
at com.cloudbees.jenkins.plugins.assurance.CloudBeesAssurance.getReport(CloudBeesAssurance.java:327)
at com.cloudbees.jenkins.plugins.assurance.EnvelopeWatch.isActivated(EnvelopeWatch.java:43)
at jenkins.management.AdministrativeMonitorsDecorator.getActiveAdministrativeMonitors(AdministrativeMonitorsDecorator.java:91)
at jenkins.management.AdministrativeMonitorsDecorator.getActiveAdministrativeMonitorsCount(AdministrativeMonitorsDecorator.java:76)
at jenkins.management.AdministrativeMonitorsDecorator.shouldDisplay(AdministrativeMonitorsDecorator.java:142)
at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
...
Environment
-
CloudBees Jenkins Enterprise - Managed controller (CJEMM)
-
CloudBees Jenkins Enterprise - Operations Center (CJEOC)
-
CloudBees Assurance Plugin 2.32.0.3 to 2.46.2.3 (excluded)
Related Issue(s)
-
CJP-6953 - Several BLOCKED slow-requests while the Beekeeper report is being generated
Explanation
The release of version 2.32.0.3 of the CloudBees Assurance Plugin introduced a performance issue when Beekeeper parses the Update Center metadata to refresh the Beekeeper report. This can slow down a Jenkins instance and can lead to serious performance issues.
Resolution
The problem has been fixed in the version 2.46.2.3 of the CloudBees Assurance Plugin. The recommended solution is to upgrade Jenkins to the latest rolling release.
Workaround
If upgrading Jenkins is not an option and you are suffering from this issue, the workaround would be to disable the administrative monitor com.cloudbees.jenkins.plugins.assurance.EnvelopeWatch in Manage Jenkins > Configure System > Administrative monitors configuration.