After moving a job, symlinks for folders became actual folders

Issue

Moving a job into a folder doesn’t preserve Unix symbolic links like lastFailedBuild, lastSuccessfulBuild, lastUnsuccessfulBuild, lastStableBuild, lastUnstableBuild in the builds folder. Links are replaced by real folders (ref: CJP-3875).

After the move operation, you can see this exception in Jenkins logs:

Nov 25, 2015 3:10:50 PM jenkins.model.PeepholePermalink updateCache
WARNING: Failed to update hudson.model.FreeStyleProject@19dcf7e4[folder/bob] lastUnsuccessfulBuild permalink for folder/bob #1
java.nio.file.DirectoryNotEmptyException: /home/dudu/tmp/jenkins-home/jobs/folder/jobs/bob/builds/lastUnsuccessfulBuild
	at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
	at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
	at java.nio.file.Files.delete(Files.java:1126)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at hudson.Util.deleteFile(Util.java:247)
	at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:113)
	at jenkins.model.PeepholePermalink.writeSymlink(PeepholePermalink.java:200)
	at jenkins.model.PeepholePermalink.updateCache(PeepholePermalink.java:150)

After the move, Jenkins is not able to update symlink and fails with this exception:

Nov 25, 2015 3:15:14 PM jenkins.model.PeepholePermalink updateCache
WARNING: Failed to update hudson.model.FreeStyleProject@19dcf7e4[folder/bob] lastUnsuccessfulBuild permalink for folder/bob #2
java.nio.file.DirectoryNotEmptyException: /home/dudu/tmp/jenkins-home/jobs/folder/jobs/bob/builds/lastUnsuccessfulBuild
	at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
	at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
	at java.nio.file.Files.delete(Files.java:1126)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Environment

CloudBees Operations Center
CloudBees Jenkins Enterprise
Linux

Resolution

The new version 1.8.2 of the operations-center-context plugin contains the fix on broken symbolic links after a move/copy/promote operation.

The new version avoids to create new erroneous directories but it won’t repair existing broken directories.

To fix your Jenkins client controller instance (on Linux):

Upgrade the operations-center-context plugin to a version >= 1.8.2
Stop your jenkins instance
Manually delete all erroneous directories from the disk under ${JENKINS_HOME}/jobs/.
You can find them in your logs with grep --no-filename --color -r -s "permalink" ${JENKINS_HOME}/logs/
You can also find them on the filesystem with find ${JENKINS_HOME}/jobs/ -type d $ -name lastFailedBuild -o -name lastSuccessfulBuild -o -name lastUnsuccessfulBuild -o -name lastStableBuild -o -name lastUnstableBuild -o -name lastFailed -o -name lastSuccessful -o -name lastUnsuccessful -o -name lastStable -o -name lastUnstable $
You can delete all of them (ALWAYS DO A BACKUP BEFORE) with find ${JENKINS_HOME}/jobs/ -type d $ -name lastFailedBuild -o -name lastSuccessfulBuild -o -name lastUnsuccessfulBuild -o -name lastStableBuild -o -name lastUnstableBuild -o -name lastFailed -o -name lastSuccessful -o -name lastUnsuccessful -o -name lastStable -o -name lastUnstable $ -exec rm -rfv "{}" \;
Restart jenkins which will recreate the correct symbolic links.

To fix your Jenkins client controller instance (on Windows):

Upgrade the operations-center-context plugin to a version >= 1.8.2
Stop your jenkins instance
Manually delete all erroneous directories from the disk under %JENKINS_HOME%\jobs\.

You can find them on the filesystem with

  cd "%JENKINS_HOME%\jobs"

  dir lastFailedBuild /ad /s /b > dirs-to-delete.list
  dir lastSuccessfulBuild /ad /s /b >> dirs-to-delete.list
  dir lastUnsuccessfulBuild /ad /s b/ >> dirs-to-delete.list
  dir lastStableBuild /ad /s b/ >> dirs-to-delete.list
  dir lastUnstableBuild /ad /s b/ >> dirs-to-delete.list
  dir lastFailed /ad /s b/ >> dirs-to-delete.list
  dir lastSuccessful /ad /s b/ >> dirs-to-delete.list
  dir lastUnsuccessful /ad /s b/ >> dirs-to-delete.list
  dir lastStable /ad /s b/ >> dirs-to-delete.list
  dir lastUnstable /ad /s b/ >> dirs-to-delete.list

Create backups of directories with for /f %i in (dirs-to-delete.list) do echo d | xcopy /f /y %i %i.bak
Verify that a copy of the directory has been created
Delete originals with for /f %i in (dirs-to-delete.list) do rmdir /q /s %i
Restart jenkins which will recreate the correct symbolic links.