Lock Contention in RBAC PersistenceHelper.getOrCreateGroupContainer

This article references an issue that affects a product version that is no longer supported. Please verify the version listed in the article applies to your situation. If unsure, please submit a support ticket at: https://support.cloudbees.com/.

Issue

Jenkins job stops building, Jenkins becomes unresponsive and thread dumps shows a lot of BLOCKED threads like the following:

  "AtmostOneTaskExecutor[Periodic Jenkins queue maintenance] [#000000]" #00000000 daemon prio=5 os_prio=0 tid=0x0000000000000000 nid=0x000d waiting for monitor entry [0x0000000000000000]
  java.lang.Thread.State: BLOCKED (on object monitor)
      at nectar.plugins.rbac.groups.GroupContainerLocator$PersistenceHelper.getOrCreateGroupContainer(GroupContainerLocator.java:202)
      - waiting to lock <0x00000000000000001> (a com.cloudbees.hudson.plugins.folder.properties.FolderProxyGroupContainer$GroupContainerLocatorImpl$1)
      at com.cloudbees.hudson.plugins.folder.properties.FolderProxyGroupContainer$GroupContainerLocatorImpl.get(FolderProxyGroupContainer.java:224)
      at nectar.plugins.rbac.groups.GroupContainerLocator.locate(GroupContainerLocator.java:118)
      at nectar.plugins.rbac.strategy.RoleMatrixAuthorizationStrategyImpl.getACL(RoleMatrixAuthorizationStrategyImpl.java:180)
      at hudson.model.AbstractItem.getACL(AbstractItem.java:601)
      [...]

Environment

CloudBees CI (CloudBees Core) < 2.249.3.1
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller < 2.249.3.1
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center < 2.249.3.1
CloudBees CI (CloudBees Core) on traditional platforms - Client controller < 2.249.3.1
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center < 2.249.3.1
CloudBees Role-Based Access Controller plugin <= 5.42

Related Issue(s)

CTR-2641: thread contention on PersistenceHelper.getOrCreateGroupContainer
CTR-430: Fix nectar-rbac with doConfigDotXml()

Explanation

This is a known issue in the CloudBees Role-Based Access Controller plugin where the PersistenceHelper.getOrCreateGroupContainer method is synchronized causing a single point of contention for all lookups. In very active instances, this can cause severe performance issues, block the queue and make the Jenkins controller unresponsive.

Workaround

There is no other workaround than restarting the Jenkins controller.

Resolution

Upgrade CloudBees CI to version 2.249.3.1 or later.