Issue
Moving a job into a folder doesn’t preserve Unix symbolic links like lastFailedBuild, lastSuccessfulBuild, lastUnsuccessfulBuild, lastStableBuild, lastUnstableBuild in the builds folder.
Links are replaced by real folders (ref: CJP-3875).
After the move operation, you can see this exception in Jenkins logs:
Nov 25, 2015 3:10:50 PM jenkins.model.PeepholePermalink updateCache WARNING: Failed to update hudson.model.FreeStyleProject@19dcf7e4[folder/bob] lastUnsuccessfulBuild permalink for folder/bob #1 java.nio.file.DirectoryNotEmptyException: /home/dudu/tmp/jenkins-home/jobs/folder/jobs/bob/builds/lastUnsuccessfulBuild at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) at java.nio.file.Files.delete(Files.java:1126) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at hudson.Util.deleteFile(Util.java:247) at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:113) at jenkins.model.PeepholePermalink.writeSymlink(PeepholePermalink.java:200) at jenkins.model.PeepholePermalink.updateCache(PeepholePermalink.java:150)
After the move, Jenkins is not able to update symlink and fails with this exception:
Nov 25, 2015 3:15:14 PM jenkins.model.PeepholePermalink updateCache WARNING: Failed to update hudson.model.FreeStyleProject@19dcf7e4[folder/bob] lastUnsuccessfulBuild permalink for folder/bob #2 java.nio.file.DirectoryNotEmptyException: /home/dudu/tmp/jenkins-home/jobs/folder/jobs/bob/builds/lastUnsuccessfulBuild at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) at java.nio.file.Files.delete(Files.java:1126) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Resolution
The new version 1.8.2 of the operations-center-context plugin contains the fix on broken symbolic links after a move/copy/promote operation.
The new version avoids to create new erroneous directories but it won’t repair existing broken directories.
To fix your Jenkins client controller instance (on Linux):
-
Upgrade the
operations-center-contextplugin to a version >= 1.8.2 -
Stop your jenkins instance
-
Manually delete all erroneous directories from the disk under
${JENKINS_HOME}/jobs/. -
You can find them in your logs with
grep --no-filename --color -r -s "permalink" ${JENKINS_HOME}/logs/ -
You can also find them on the filesystem with
find ${JENKINS_HOME}/jobs/ -type d \( -name lastFailedBuild -o -name lastSuccessfulBuild -o -name lastUnsuccessfulBuild -o -name lastStableBuild -o -name lastUnstableBuild -o -name lastFailed -o -name lastSuccessful -o -name lastUnsuccessful -o -name lastStable -o -name lastUnstable \) -
You can delete all of them (ALWAYS DO A BACKUP BEFORE) with
find ${JENKINS_HOME}/jobs/ -type d \( -name lastFailedBuild -o -name lastSuccessfulBuild -o -name lastUnsuccessfulBuild -o -name lastStableBuild -o -name lastUnstableBuild -o -name lastFailed -o -name lastSuccessful -o -name lastUnsuccessful -o -name lastStable -o -name lastUnstable \) -exec rm -rfv "{}" \; -
Restart jenkins which will recreate the correct symbolic links.
To fix your Jenkins client controller instance (on Windows):
-
Upgrade the
operations-center-contextplugin to a version >= 1.8.2 -
Stop your jenkins instance
-
Manually delete all erroneous directories from the disk under
%JENKINS_HOME%\jobs\. -
You can find them on the filesystem with
cd "%JENKINS_HOME%\jobs" dir lastFailedBuild /ad /s /b > dirs-to-delete.list dir lastSuccessfulBuild /ad /s /b >> dirs-to-delete.list dir lastUnsuccessfulBuild /ad /s b/ >> dirs-to-delete.list dir lastStableBuild /ad /s b/ >> dirs-to-delete.list dir lastUnstableBuild /ad /s b/ >> dirs-to-delete.list dir lastFailed /ad /s b/ >> dirs-to-delete.list dir lastSuccessful /ad /s b/ >> dirs-to-delete.list dir lastUnsuccessful /ad /s b/ >> dirs-to-delete.list dir lastStable /ad /s b/ >> dirs-to-delete.list dir lastUnstable /ad /s b/ >> dirs-to-delete.list
-
Create backups of directories with
for /f %i in (dirs-to-delete.list) do echo d | xcopy /f /y %i %i.bak -
Verify that a copy of the directory has been created
-
Delete originals with
for /f %i in (dirs-to-delete.list) do rmdir /q /s %i -
Restart jenkins which will recreate the correct symbolic links.