Issue
-
Jenkins' builds and navigation is slowed down, and a thread dump shows several threads
BLOCKED
waiting forjenkins.fingerprints.FileFingerprintStorage
:[...] at jenkins.fingerprints.FileFingerprintStorage.save(FileFingerprintStorage.java:125) - waiting to lock <0x00007f9b65846a98> (a jenkins.fingerprints.FileFingerprintStorage) [...]
Environment
-
CloudBees CI (CloudBees Core) >= 2.249.1.2
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller >= 2.249.1.2
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center >= 2.249.1.2
-
CloudBees CI (CloudBees Core) on traditional platforms - Client controller >= 2.249.1.2
-
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center >= 2.249.1.2
-
CloudBees Jenkins Platform - Client controller >= 2.249.1.2
-
CloudBees Jenkins Platform - Operations Center >= 2.249.1.2
-
CloudBees Jenkins Distribution >= 2.249.1.2
-
Jenkins LTS >= 2.249.1
Related Issue(s)
-
JENKINS-62345: Introduce External Fingerprint Storage API in core
-
JENKINS-64670: FileFingerprintStorage performance
Explanation
Jenkins LTS 2.249.1 introduces an API to enable pluggable storage for Jenkins Fingerprints.
The default implementation of that API uses the File System under $JENKINS_HOME/fingerprints
like previous version. However, a bug - JENKINS-64670 - was introduced that prevent several Fingerprints to be saved in parallel. All Fingerprint saves are blocked by a single object lock on the jenkins.fingerprints.FileFingerprintStorage
.
This can cause severe performances issues when used with plugin that produce lots of fingerprints. For example the Maven Plugin and the Credentials plugin when tracking credentials usage.
Resolution
A solution has already been merged in Jenkins Core and will likely be available in the next Jenkins LTS. The March 2021 release of CloudBees CI will be based on the new LTS.
For impacted instances, it is recommended to stay up to date and prepare for this release.
Workaround
There is no workaround for the lock contention.
However, if impacted it is likely that the fingerprinting for Credentials usage contributes greatly to the problem. In which case it is recommended to disable it, see Performance issues caused by the Fingerprinting of Credentials Usage.