Lock Contention in RBAC PersistenceHelper.getOrCreateGroupContainer

Article ID:4411473583259
1 minute readKnowledge base
This article references an issue that affects a product version that is no longer supported. Please verify the version listed in the article applies to your situation. If unsure, please submit a support ticket at: https://support.cloudbees.com/.

Issue

  • Jenkins job stops building, Jenkins becomes unresponsive and thread dumps shows a lot of BLOCKED threads like the following:

      "AtmostOneTaskExecutor[Periodic Jenkins queue maintenance] [#000000]" #00000000 daemon prio=5 os_prio=0 tid=0x0000000000000000 nid=0x000d waiting for monitor entry [0x0000000000000000]
      java.lang.Thread.State: BLOCKED (on object monitor)
          at nectar.plugins.rbac.groups.GroupContainerLocator$PersistenceHelper.getOrCreateGroupContainer(GroupContainerLocator.java:202)
          - waiting to lock <0x00000000000000001> (a com.cloudbees.hudson.plugins.folder.properties.FolderProxyGroupContainer$GroupContainerLocatorImpl$1)
          at com.cloudbees.hudson.plugins.folder.properties.FolderProxyGroupContainer$GroupContainerLocatorImpl.get(FolderProxyGroupContainer.java:224)
          at nectar.plugins.rbac.groups.GroupContainerLocator.locate(GroupContainerLocator.java:118)
          at nectar.plugins.rbac.strategy.RoleMatrixAuthorizationStrategyImpl.getACL(RoleMatrixAuthorizationStrategyImpl.java:180)
          at hudson.model.AbstractItem.getACL(AbstractItem.java:601)
          [...]
  • CTR-2641: thread contention on PersistenceHelper.getOrCreateGroupContainer

  • CTR-430: Fix nectar-rbac with doConfigDotXml()

Explanation

This is a known issue in the CloudBees Role-Based Access Controller plugin where the PersistenceHelper.getOrCreateGroupContainer method is synchronized causing a single point of contention for all lookups. In very active instances, this can cause severe performance issues, block the queue and make the Jenkins controller unresponsive.

Workaround

There is no other workaround than restarting the Jenkins controller.

Resolution

Upgrade CloudBees CI to version 2.249.3.1 or later.