Issue
-
After upgrading the kubernetes to version 1.18.0 or later, some builds are failing on provisioned agent with issues such as
permission denied
orfile or directory does not exist
and pointing to/home/jenkins/
directory.
Environment
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
-
CloudBees CI (CloudBees Core) on traditional platforms - Client controller
-
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
-
Kubernetes Plugin 1.18.0 or later
Related Issues
-
JENKINS-58705 (breaking change)
Explanation
In an Openshift environment, the kubernetes plugin enforce a default HOME
pod environment variable that points to /home/jenkins
, to support Creating images documentation.
This was working fine when the working directory was /home/jenkins
. Thw working directory is mounted as an emptyDir
volume and the running user has write permission in it.
Since Kubernetes Plugin version 1.18.0, the default working directory of the pod containers was changed from /home/jenkins
to /home/jenkins/agent
. But the default HOME
environment variable enforcement is still pointing to /home/jenkins
. The impact of this change is that if pod container images do not have a /home/jenkins
directory with sufficient permissions for the running user, builds will fail to do anything directly under their HOME
directory, /home/jenkins
.
Therefore this is a breaking change in Openshift when using pod templates that have multiple containers or override the jnlp
container with a custom image that does not have an existing /home/jenkins
directory.
Resolution
There are different workaround to that problem:
Change the default HOME
variable
The simplest and preferred workaround is to add the system property -Dorg.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder.defaultHome=/home/jenkins/agent
on Jenkins startup. This requires a restart.
This workaround will reflect the behavior of kubernetes plugin pre-1.18.0 but on the new working directory /home/jenkins/agent
Use /home/jenkins
as the working directory
A workaround is to change the working directory of pod containers back to /home/jenkins
. This workaround is only possible when using YAML to define agent pod templates (see JENKINS-60977).
Prepare images for Jenkins
A workaround could be to ensure that the images used in agent pods have a /home/jenkins
directory that is owned by the root
group and writable by the root
group as mentioned in OpenShift Container Platform-specific guidelines.