Issue
-
I want to add global configuration files or directory such as
~/.gitconfig
,~/.ssh
,~/.npmrc
,~/.docker/config.json
, etc… to agent pods -
I have added configuration files under
/home/jenkins
when building my docker image but the files are not in the container when the kubernetes agent starts
Explanation
The Kubernetes Plugin create an emptyDir volume to share the build workspace across the containers inside Agent pods.
Before version 1.18 of the plugin:
-
the working directory is defaulted to
/home/jenkins
, the home directory of the Jenkins user
This enforcement prevents adding any files to /home/jenkins
since it is mounted from an emptyDir
volume. And it implies that:
-
existing files in the docker image under
/home/jenkins
by default, are wipe out when the container start -
mounting a single file under
/home/jenkins
withsubPath
is a known limitation, see kubernetes #61563 for more details.
Changing the working directory seems like a solution but another enforcement comes into play
-
the
$HOME
environment variable is set to the working directory location
This causes global tools to looks for configuration to the wrong place (and again the emptyDir
). For example if the working directory is set to /home/jenkins/agent
, a tool like ssh
would looks for private keys under /home/jenkins/agent/.ssh
.
Since version 1.18, the working directory and the jenkins user directory are distinct:
-
default working directory is
/home/jenkins/agent
-
default
$HOME
is/home/jenkins
Which enables to preserve existing files under /home/jenkins
and add more via configmap and secrets.
Resolution
A good solution in kubernetes is to create Configmaps (and/or Secrets for sensitive information) that contain such configuration and then mount them to the agent pod as volumes.
In the following sections we provide example with a Secret and a Configmap:
-
The Configmap
my-git-config
has been created with the commandkubectl create configmap jenkins-git-config --from-file=/home/test/.gitconfig
-
The Secret
my-docker-config
has been created with the commandkubectl create secret generic my-docker-config --from-file=.dockerconfigjson=/home/test/.docker/config.json --type=kubernetes.io/dockerconfigjson
Kubernetes Plugin 1.18.0 and later
Since Kubernetes 1.18.0, the configuration can simply be mounted to the pod container(s).
The following is an example of a Pod YAML configuration that injects a .gitconfig
from a configmap and a .docker/config.json
from secret in the agent’s user home directory:
kind: Pod metadata: labels: example: "gitconfig" spec: containers: - name: jnlp volumeMounts: - name: jenkins-git-config-volume mountPath: /home/jenkins/.gitconfig subPath: .gitconfig - name: jenkins-docker-config-volume mountPath: /home/jenkins/.docker volumes: - name: jenkins-git-config-volume configMap: name: jenkins-git-config items: - key: .gitconfig path: .gitconfig - name: jenkins-docker-config-volume secret: secretName: my-docker-config items: - key: .dockerconfigjson path: config.json
Following is a full example in a pipeline:
pipeline { agent { kubernetes { label 'jnlpAgentWithGitConfig1-18' yaml """ kind: Pod metadata: labels: example: "gitconfig" spec: containers: - name: jnlp volumeMounts: - name: jenkins-git-config-volume mountPath: /home/jenkins/.gitconfig subPath: .gitconfig - name: jenkins-docker-config-volume mountPath: /home/jenkins/.docker volumes: - name: jenkins-git-config-volume configMap: name: jenkins-git-config items: - key: .gitconfig path: .gitconfig - name: jenkins-docker-config-volume secret: secretName: my-docker-config items: - key: .dockerconfigjson path: config.json """ } } stages { stage('Test Git config') { steps { sh "git config --global -l" } } stage('Test Docker config') { steps { sh "cat ~/.docker/config.json" } } } }
Kubernetes Plugin < 1.18.0
Before the release of Kubernetes 1.18.0, the working directory of the jnlp
container was defaulted to the user directory of the Jenkins user /home/jenkins
. This prevented to mount any file to /home/jenkins
. To workaround that problem:
-
change the working directory of all container to
/home/jenkins/agent
-
change the environment variable to
/home/jenkins
during the pipeline build
The following is an example of a Pod YAML configuration that injects a .gitconfig
file in the agent’s user home directory and set the HOME
variable to the jenkins user’s home directory /home/user
:
kind: Pod metadata: labels: example: "gitconfig" spec: containers: - name: jnlp workingDir: /home/jenkins/agent env: - name: HOME value: /home/jenkins volumeMounts: - name: jenkins-git-config-volume mountPath: /home/jenkins/.gitconfig subPath: .gitconfig - name: jenkins-docker-config-volume mountPath: /home/jenkins/.docker volumes: - name: jenkins-git-config-volume configMap: name: jenkins-git-config items: - key: .gitconfig path: .gitconfig - name: jenkins-docker-config-volume secret: secretName: my-docker-config items: - key: .dockerconfigjson path: config.json
Following is a full example in a declarative pipeline:
pipeline { agent { kubernetes { label 'jnlpAgentWithGitConfig' yaml """ kind: Pod metadata: labels: example: "gitconfig" spec: containers: - name: jnlp workingDir: /home/jenkins/agent env: - name: HOME value: /home/jenkins volumeMounts: - name: jenkins-git-config-volume mountPath: /home/jenkins/.gitconfig subPath: .gitconfig - name: jenkins-docker-config-volume mountPath: /home/jenkins/.docker volumes: - name: jenkins-git-config-volume configMap: name: jenkins-git-config items: - key: .gitconfig path: .gitconfig - name: jenkins-docker-config-volume secret: secretName: my-docker-config items: - key: .dockerconfigjson path: config.json """ } } stages { stage('Test Git config') { steps { sh "git config --global -l" } } stage('Test Docker config') { steps { sh "cat ~/.docker/config.json" } } } }