After creating a copy of the local Jenkins home lastSuccessful fails with java.nio.file.DirectoryNotEmptyException

Issue

After creating a copy of the local Jenkins home /lastSuccessful fails with java.nio.file.DirectoryNotEmptyException
You are getting a stacktrace similar to the one below:

/opt/jenkins/jobs/PACE/jobs/Scheduled_Production_Copy_LPAR6/lastSuccessful failed
java.nio.file.DirectoryNotEmptyException: /opt/jenkins/jobs/PACE/jobs/Scheduled_Production_Copy_LPAR6/lastSuccessful
at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
at java.nio.file.Files.deleteIfExists(Files.java:1118)
at sun.reflect.GeneratedMethodAccessor4202.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at hudson.Util.createSymlinkJava7(Util.java:1194)
at hudson.Util.createSymlink(Util.java:1112)
at hudson.model.Run.createSymlink(Run.java:1851)
at hudson.model.Run.updateSymlinks(Run.java:1832)
at hudson.model.Run.execute(Run.java:1743)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:89)
at hudson.model.Executor.run(Executor.java:240)

Environment

Resolution

The issue is related to the method used to copy the $JENKINS_HOME from one location to another one, which in this particular case broke the symlinks.

The recommended option, it is to rollback to a security copy of your previous $JENKINS_HOME if possible and copy it in the new location using a method which preserve the symlinks.

If this is not possible, to fix the controller where you have these erroneous directories you should follow the steps shown below:

Important Notice: You should have a current backup of your $JENKINS_HOME or you should create one before trying the method described below.

# Logon on the controller where you have the issue
ssh machine
# Shutdown the jenkins controller (take care that nothing is running)
sudo service jenkins stop
# Find all erroneous directories
find /opt/jenkins/jobs -type d \( -name "last*Build" -o -name "lastStable" -o -name "lastSuccessful" \)
# Review the list of erroneous directories
# Rename them
find /opt/jenkins/jobs -type d \( -name "last*Build" -o -name "lastStable" -o -name "lastSuccessful" \) -exec mv {} {}.err \;
# Restart jenkins
sudo service jenkins start

Once that you have verified that everything is working correctly, you should delete the faulty directories following the steps shown below:

# Delete them
find /opt/jenkins/jobs -type d -name "*.err" -exec rm -rf {} \;

Tested product/plugin versions

Jenkins LTS