Lock contention caused by introduction of the External Fingerprinting Storage API

Article ID:360056051131
2 minute readKnowledge base

Issue

  • Jenkins' builds and navigation is slowed down, and a thread dump shows several threads BLOCKED waiting for jenkins.fingerprints.FileFingerprintStorage:

        [...]
        at jenkins.fingerprints.FileFingerprintStorage.save(FileFingerprintStorage.java:125)
          - waiting to lock <0x00007f9b65846a98> (a jenkins.fingerprints.FileFingerprintStorage)
        [...]

Explanation

Jenkins LTS 2.249.1 introduces an API to enable pluggable storage for Jenkins Fingerprints.

The default implementation of that API uses the File System under $JENKINS_HOME/fingerprints like previous version. However, a bug - JENKINS-64670 - was introduced that prevent several Fingerprints to be saved in parallel. All Fingerprint saves are blocked by a single object lock on the jenkins.fingerprints.FileFingerprintStorage.

This can cause severe performances issues when used with plugin that produce lots of fingerprints. For example the Maven Plugin and the Credentials plugin when tracking credentials usage.

Resolution

A solution has already been merged in Jenkins Core and will likely be available in the next Jenkins LTS. The March 2021 release of CloudBees CI will be based on the new LTS.

For impacted instances, it is recommended to stay up to date and prepare for this release.

Workaround

There is no workaround for the lock contention.

However, if impacted it is likely that the fingerprinting for Credentials usage contributes greatly to the problem. In which case it is recommended to disable it, see Performance issues caused by the Fingerprinting of Credentials Usage.