Pipelines and Multibranch Pipelines are failing due to 'java.nio.file.AccessDeniedException'

Issue

If you are running CloudBees Core Traditional in High Availability (active/passive) (specially on Windows), you may eventually get this error in your Pipeline/Multibranch builds

[Pipeline] End of Pipeline
Also:   java.nio.file.AccessDeniedException:
            C:\JenkinsHome\jobs\FOLDER1\jobs\FOLDER2\jobs\Multibranch1\branches\feature-56\builds\4\atomic7796248186654507297tmp
            -> C:\JenkinsHome\jobs\FOLDER1\jobs\FOLDER2\jobs\Multibranch1\branches\feature-56\builds\4\build.xml
        at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
        at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
        at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
        at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
        at java.nio.file.Files.move(Unknown Source)
        at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:191)
        at hudson.XmlFile.write(XmlFile.java:198)
        at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:30)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1140)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
java.nio.file.AccessDeniedException: C:\JenkinsHome\jobs\FOLDER1\jobs\FOLDER2\jobs\Multibranch1\branches\feature-56\builds\4\atomic7796248186654507297tmp -> C:\JenkinsHome\jobs\FOLDER1\jobs\FOLDER2\jobs\Multibranch1\branches\feature-56\builds\4\build.xml
    at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
    at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
    at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
    at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
    at java.nio.file.Files.move(Unknown Source)
    at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:206)
    at hudson.XmlFile.write(XmlFile.java:198)
    at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:30)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1140)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
    at org.jenkinsci.plugins.pipeline.modeldefinition.Utils.updateRunAndJobActions(Utils.groovy:293)
    at org.jenkinsci.plugins.pipeline.modeldefinition.Utils$updateRunAndJobActions$4.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
    at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.call(ModelInterpreter.groovy:67)
    at WorkflowScript.run(WorkflowScript:22)
    at ___cps.transform___(Native Method)
...
Finished: FAILURE

The error may appear randomly in some builds under certain unknown race conditions.

Environment

Resolution

Unfortunately, most of the time the underlying problem is on the filesystem. Any High Availability installation requires that the $JENKINS_HOME is located in a shared volume. It’s important that the shared volume (depending on the architecture) is properly tuned and avoid to lock files.

Workaround

Sometimes it can help to handle the previous error reducing the amount of I/O operations done by the build.

To do that, go to Manage Jenkins > Configure System > Pipeline Speed/Durability Settings, select the option Performance-optimized: much faster (requires clean shutdown to save running pipelines) and saves the changes.

That behavior can be overridden:

In a Pipeline job by setting the property Pipeline speed/durability override
In a Multibranch job by adding the Property strategy: Pipeline branch speed/durability override

That behaviour can’t be overriden in GitHub/Bitbucket Organization jobs.

References

Scaling Pipelines