This article references an issue that affects a product version that is no longer supported. Please verify the version listed in the article applies to your situation. If unsure, please submit a support ticket at: https://support.cloudbees.com/. |
Issue
-
Jenkins job stops building, Jenkins becomes unresponsive and thread dumps shows a lot of BLOCKED threads like the following:
"AtmostOneTaskExecutor[Periodic Jenkins queue maintenance] [#000000]" #00000000 daemon prio=5 os_prio=0 tid=0x0000000000000000 nid=0x000d waiting for monitor entry [0x0000000000000000] java.lang.Thread.State: BLOCKED (on object monitor) at nectar.plugins.rbac.groups.GroupContainerLocator$PersistenceHelper.getOrCreateGroupContainer(GroupContainerLocator.java:202) - waiting to lock <0x00000000000000001> (a com.cloudbees.hudson.plugins.folder.properties.FolderProxyGroupContainer$GroupContainerLocatorImpl$1) at com.cloudbees.hudson.plugins.folder.properties.FolderProxyGroupContainer$GroupContainerLocatorImpl.get(FolderProxyGroupContainer.java:224) at nectar.plugins.rbac.groups.GroupContainerLocator.locate(GroupContainerLocator.java:118) at nectar.plugins.rbac.strategy.RoleMatrixAuthorizationStrategyImpl.getACL(RoleMatrixAuthorizationStrategyImpl.java:180) at hudson.model.AbstractItem.getACL(AbstractItem.java:601) [...]
Environment
-
CloudBees CI (CloudBees Core) < 2.249.3.1
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller < 2.249.3.1
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center < 2.249.3.1
-
CloudBees CI (CloudBees Core) on traditional platforms - Client controller < 2.249.3.1
-
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center < 2.249.3.1
Related Issue(s)
-
CTR-2641: thread contention on PersistenceHelper.getOrCreateGroupContainer
-
CTR-430: Fix nectar-rbac with doConfigDotXml()
Explanation
This is a known issue in the CloudBees Role-Based Access Controller plugin where the PersistenceHelper.getOrCreateGroupContainer
method is synchronized causing a single point of contention for all lookups. In very active instances, this can cause severe performance issues, block the queue and make the Jenkins controller unresponsive.
Resolution
Upgrade CloudBees CI to version 2.249.3.1 or later.