Performance issues caused by the Fingerprinting of Credentials Usage

Article ID:360055304192
2 minute readKnowledge base

Issue

  • Jenkins shows disk space issues although the disk holding $JENKINS_HOME is not full:

      java.io.IOException: No space left on device
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createTempFile(File.java:2024)
    at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:142)

  • Jenkins' builds and navigation is slowed down, and a thread dump shows several threads BLOCKED or WAITING like the following:

        [...]
    at hudson.XmlFile.write(XmlFile.java:193)
    at hudson.model.Fingerprint.save(Fingerprint.java:1307)
    at hudson.model.Fingerprint.save(Fingerprint.java:1251)
    - eliminated <0x00007f2ec6c47108> (a hudson.model.Fingerprint)
    at hudson.model.Fingerprint.add(Fingerprint.java:1029)
    - eliminated <0x00007f2ec6c47108> (a hudson.model.Fingerprint)
    at hudson.model.Fingerprint.addFor(Fingerprint.java:1021)
    - locked <0x00007f2ec6c47108> (a hudson.model.Fingerprint)
    at com.cloudbees.plugins.credentials.CredentialsProvider.trackAll(CredentialsProvider.java:1457)
    at com.cloudbees.plugins.credentials.CredentialsProvider.track(CredentialsProvider.java:1421)
    [...]

Environment

Related Issue(s)

Explanation

The fingerprinting of credentials used for tracking credentials usage can cause a very large number of files to be created under $JENKINS_HOME/fingerprints. To check if you are out of inodes, run df -i and check the %iused column for any mounts at 100% used.

  • A command like find $JENKINS_HOME/fingerprints -name *.xml -ctime -1 -exec tar -rf recent_inodes.tar {} \; can be used to find all fingerprint files created within the last day.

  • A command like find $JENKINS_HOME/fingerprints -name *.xml -mtime -1 -exec tar -rf recent_inodes.tar {} \; can be used to find all fingerprint files modified within the last day.

In most cases, it causes inodes usage spikes. It could well reveals itself as lock contention on jenkins.fingerprints.FileFingerprintStorage.save from com.cloudbees.plugins.credentials.CredentialsProvider.trackAll.

Resolution

If impacted by this issue, make sure you run version 2.3.5 or later of the Credentials plugin. Then disable the fingerprinting for credentials tracking by executing the script com.cloudbees.plugins.credentials.CredentialsProvider.FINGERPRINT_ENABLED=false in Manage Jenkins  Script Console. To make the change persistent, add the system property -DCredentialsProvider.fingerprintEnabled=false on start up. See How to add Java arguments to Jenkins.

By disabling the fingerprinting of credentials, the functionality that tracks credentials usage is disabled (this functionality allows a user with access to the credentials page to check what job / build has used a specific credentials recently).

Troubleshooting