Global Build Discarders causes exceptions after Upgrading to 2.222.x

Issue

After upgrading CloudBees Core to 2.222.x, the Jenkins logs shows a lot of exception like the following:

  Also: java.nio.file.NoSuchFileException: <buildPath> -> <buildPath>
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at hudson.model.Run.delete(Run.java:1621)
    at hudson.tasks.LogRotator.perform(LogRotator.java:166)
  jenkins.util.io.CompositeIOException: Failed to rotate logs for [<buildDisplayName>]
    at hudson.tasks.LogRotator.perform(LogRotator.java:223)
    at hudson.model.Job.logRotate(Job.java:469)
    at jenkins.model.JobGlobalBuildDiscarderStrategy.apply(JobGlobalBuildDiscarderStrategy.java:54)
    at jenkins.model.BackgroundGlobalBuildDiscarder.lambda$processJob$0(BackgroundGlobalBuildDiscarder.java:67)
    at java.lang.Iterable.forEach(Iterable.java:75)
    at jenkins.model.BackgroundGlobalBuildDiscarder.processJob(BackgroundGlobalBuildDiscarder.java:61)
    at jenkins.model.GlobalBuildDiscarderListener.onFinalized(GlobalBuildDiscarderListener.java:49)
    at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:255)
    at hudson.model.Run.onEndBuilding(Run.java:2018)
    [...]▼

Environment

CloudBees CI (CloudBees Core) >= 2.222.1.2 < 2.249.1.2
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller >= 2.222.1.2 < 2.249.1.2
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center >= 2.222.1.2 < 2.249.1.2
CloudBees CI (CloudBees Core) on traditional platforms - Client controller >= 2.222.1.2 < 2.249.1.2
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center >= 2.222.1.2 < 2.249.1.2

Related Issues

JENKINS-61687: Exceptions due to race condition during build deletion (fixed in Jenkins LTS 2.249.1)

Explanation

Jenkins Core 2.222 introduce Global Build Discarders as mentioned in the release notes. Those strategies apply the discard build policy of jobs periodically every hour but also whenever a build completes - like the job build discarders do.

Whenever a build finish, the build discarder may be triggered twice for the same job and is subjected to a race condition in the build deletion logic. The race condition is captured by JENKINS-61687. Jenkins might try to delete the same build twice causing the exception shown above.

In most cases, the impact is minor. However if this happens very frequently - like in large and active environments - it can spam the Jenkins logs and may well cause performance issues on the controller.

Resolution

Upgrade CloudBees CI to 2.249.1.2 or later.

Workaround

The workaround is to delete the Global Build Discarder Policies.

This can be done from the UI:

Go to Manage Jenkins Configure System
Navigate to "Global Build Discarders"
Delete any of the "Project Build Discarder" or "Specific Build Discarder" that currently exist
Save

This can also be done via a Groovy script like the following:

jenkins.model.GlobalBuildDiscarderConfiguration.get().configuredBuildDiscarders.clear()
jenkins.model.GlobalBuildDiscarderConfiguration.get().save()▼

Tested product/plugin versions

The latest update of this article has been tested and verified with:

CloudBees CI (CloudBees Core) 2.235.2.3