Issue
After upgrading my instance, some of my jobs are suddenly failing with the following stack trace:
> git config remote.origin.url $SOME_GIT_URL # timeout=10 ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from $SOME_GIT_URL at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:1002) [...] Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url $SOME_GIT_URL" returned status code 128: stdout: stderr: fatal: not in a git directory at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2734) [...] ERROR: Error fetching remote repo 'origin' ERROR: Maximum checkout retry attempts reached, aborting org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: WorkflowScript: Loading libraries failed 1 error at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:309) [...] Finished: FAILURE
Context
First of all, is worth noting this issue is not a Jenkins-related issue but a Git-related issue. In the Git release 2.35.2 was introduced a couple of fixes for some security vulnerabilities, including the one that causes this scenario: CVE-2022-24765.
Starting on release 2.35.2, Git now checks if the User that is currently trying to access a particular Git directory is the same User that is defined as the Owner of that Git directory itself. If is not, the exception shared above will be triggered.
Regarding how this Git issue affects CloudBees CI, it can arise because of one of the following three possibilities:
1) Using the default images on a Modern Platform, you have upgraded from release 2.387.3.5 or lesser, to release 2.401.1.3 or higher. As previously mentioned, this additional Git check was introduced in release 2.35.2, and since 2.401.1.3 the Git client used in the default images was upgraded to a higher release than 2.35.2.
2) Using custom images on a Modern Platform, you have upgraded your Git client release to 2.35.2 or higher.
3) Using the Traditional Platform, you have upgraded your Jenkins host machine Git client to release 2.35.2 or higher.
Workaround
This scenario is caused by an additional Git check of the permissions as described above. Nevertheless, you can execute the command below to add a particular directory as a safe.directory
git config --global --add safe.directory 'DIRECTORY_TO_MARK_SAFE'
Or you can add all your filesystem at once as a safe.directory using the following command:
git config --global --add safe.directory '*'
| This workaround should only be a temporary, we recommend following the Resolution steps below instead |
When you add a directory to the Git safe.directory list you have to be aware that you are bypassing the new Git mechanism introduced on release 2.35.2 that forces the match between the User that is trying to access a particular Git directory and the Owner of the Git directory itself. Our recommendation is to follow the indications in the Resolution section and use the described Git safe.directory workaround as a temporal patch while implementing the indications below.
Resolution
This unexpected behavior is caused by a mismatch between the $JENKINS_USER, which is the User that is running Jenkins and that is trying to access a Git directory, and the Owner of the Git directory that this $JENKINS_USER is trying to access. For example, the $JENKINS_USER may be called cloudbees-core-cm and is trying to perform a git checkout within the directory $JENKINS_HOME/workspace, but, the output of the command ls -la $JENKINS_HOME reveals that, indeed, the Owner of the $JENKINS_HOME/workspace directory is defined to root and not cloudbees-core-cm.
The best approach is to define the same User for both the $JENKINS_USER and the Owner of the entire $JENKINS_HOME to avoid any possible future related incident. To do so, you can execute the command below to override the ownership of the entire $JENKINS_HOME:
chown -R $JENKINS_USER:$JENKINS_USER $YOUR_JENKINS_HOME
If running in Kubernetes, and you are having problems modifying the permissions, review CloudBees CI on modern cloud platforms StorageClass UID mismatch.