Build fail on Openshift agent after upgrading the kubernetes plugin to 1.18.0 or later

Article ID:360039311112
2 minute readKnowledge base

Issue

  • After upgrading the kubernetes to version 1.18.0 or later, some builds are failing on provisioned agent with issues such as permission denied or file or directory does not exist and pointing to /home/jenkins/ directory.

Explanation

In an Openshift environment, the kubernetes plugin enforce a default HOME pod environment variable that points to /home/jenkins, to support Creating images documentation.

This was working fine when the working directory was /home/jenkins. Thw working directory is mounted as an emptyDir volume and the running user has write permission in it.

Since Kubernetes Plugin version 1.18.0, the default working directory of the pod containers was changed from /home/jenkins to /home/jenkins/agent. But the default HOME environment variable enforcement is still pointing to /home/jenkins. The impact of this change is that if pod container images do not have a /home/jenkins directory with sufficient permissions for the running user, builds will fail to do anything directly under their HOME directory, /home/jenkins.

Therefore this is a breaking change in Openshift when using pod templates that have multiple containers or override the jnlp container with a custom image that does not have an existing /home/jenkins directory.

Resolution

There are different workaround to that problem:

Change the default HOME variable

The simplest and preferred workaround is to add the system property -Dorg.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder.defaultHome=/home/jenkins/agent on Jenkins startup. This requires a restart.

This workaround will reflect the behavior of kubernetes plugin pre-1.18.0 but on the new working directory /home/jenkins/agent

Use /home/jenkins as the working directory

A workaround is to change the working directory of pod containers back to /home/jenkins. This workaround is only possible when using YAML to define agent pod templates (see JENKINS-60977).

Prepare images for Jenkins

A workaround could be to ensure that the images used in agent pods have a /home/jenkins directory that is owned by the root group and writable by the root group as mentioned in OpenShift Container Platform-specific guidelines.