How to add global configuration to Kubernetes Agents

Issue

I want to add global configuration files or directory such as ~/.gitconfig, ~/.ssh, ~/.npmrc, ~/.docker/config.json, etc… to agent pods
I have added configuration files under /home/jenkins when building my docker image but the files are not in the container when the kubernetes agent starts

Environment

Related Issues

JENKINS-58705

Explanation

The Kubernetes Plugin create an emptyDir volume to share the build workspace across the containers inside Agent pods.

Before version 1.18 of the plugin:

the working directory is defaulted to /home/jenkins, the home directory of the Jenkins user

This enforcement prevents adding any files to /home/jenkins since it is mounted from an emptyDir volume. And it implies that:

existing files in the docker image under /home/jenkins by default, are wipe out when the container start
mounting a single file under /home/jenkins with subPath is a known limitation, see kubernetes #61563 for more details.

Changing the working directory seems like a solution but another enforcement comes into play

the $HOME environment variable is set to the working directory location

This causes global tools to looks for configuration to the wrong place (and again the emptyDir). For example if the working directory is set to /home/jenkins/agent, a tool like ssh would looks for private keys under /home/jenkins/agent/.ssh.

Since version 1.18, the working directory and the jenkins user directory are distinct:

default working directory is /home/jenkins/agent
default $HOME is /home/jenkins

Which enables to preserve existing files under /home/jenkins and add more via configmap and secrets.

Resolution

A good solution in kubernetes is to create Configmaps (and/or Secrets for sensitive information) that contain such configuration and then mount them to the agent pod as volumes.

In the following sections we provide example with a Secret and a Configmap:

The Configmap my-git-config has been created with the command kubectl create configmap jenkins-git-config --from-file=/home/test/.gitconfig
The Secret my-docker-config has been created with the command kubectl create secret generic my-docker-config --from-file=.dockerconfigjson=/home/test/.docker/config.json --type=kubernetes.io/dockerconfigjson

Kubernetes Plugin 1.18.0 and later

Since Kubernetes 1.18.0, the configuration can simply be mounted to the pod container(s).

The following is an example of a Pod YAML configuration that injects a .gitconfig from a configmap and a .docker/config.json from secret in the agent’s user home directory:

kind: Pod
metadata:
  labels:
    example: "gitconfig"
spec:
  containers:
  - name: jnlp
    volumeMounts:
      - name: jenkins-git-config-volume
        mountPath: /home/jenkins/.gitconfig
        subPath: .gitconfig
      - name: jenkins-docker-config-volume
        mountPath: /home/jenkins/.docker
  volumes:
  - name: jenkins-git-config-volume
    configMap:
      name: jenkins-git-config
      items:
      - key: .gitconfig
        path: .gitconfig
  - name: jenkins-docker-config-volume
    secret:
      secretName: my-docker-config
      items:
      - key: .dockerconfigjson
        path: config.json

Following is a full example in a pipeline:

pipeline {
  agent {
    kubernetes {
      label 'jnlpAgentWithGitConfig1-18'
      yaml """
kind: Pod
metadata:
  labels:
    example: "gitconfig"
spec:
  containers:
  - name: jnlp
    volumeMounts:
      - name: jenkins-git-config-volume
        mountPath: /home/jenkins/.gitconfig
        subPath: .gitconfig
      - name: jenkins-docker-config-volume
        mountPath: /home/jenkins/.docker
  volumes:
  - name: jenkins-git-config-volume
    configMap:
      name: jenkins-git-config
      items:
      - key: .gitconfig
        path: .gitconfig
  - name: jenkins-docker-config-volume
    secret:
      secretName: my-docker-config
      items:
      - key: .dockerconfigjson
        path: config.json
"""
    }
  }
  stages {
    stage('Test Git config') {
      steps {
        sh "git config --global -l"
      }
    }
    stage('Test Docker config') {
      steps {
        sh "cat ~/.docker/config.json"
      }
    }
  }
}

Kubernetes Plugin < 1.18.0

Before the release of Kubernetes 1.18.0, the working directory of the jnlp container was defaulted to the user directory of the Jenkins user /home/jenkins. This prevented to mount any file to /home/jenkins. To workaround that problem:

change the working directory of all container to /home/jenkins/agent
change the environment variable to /home/jenkins during the pipeline build

The following is an example of a Pod YAML configuration that injects a .gitconfig file in the agent’s user home directory and set the HOME variable to the jenkins user’s home directory /home/user:

kind: Pod
metadata:
  labels:
    example: "gitconfig"
spec:
  containers:
  - name: jnlp
    workingDir: /home/jenkins/agent
    env:
    - name: HOME
      value: /home/jenkins
    volumeMounts:
      - name: jenkins-git-config-volume
        mountPath: /home/jenkins/.gitconfig
        subPath: .gitconfig
      - name: jenkins-docker-config-volume
        mountPath: /home/jenkins/.docker
  volumes:
  - name: jenkins-git-config-volume
    configMap:
      name: jenkins-git-config
      items:
      - key: .gitconfig
        path: .gitconfig
  - name: jenkins-docker-config-volume
    secret:
      secretName: my-docker-config
      items:
      - key: .dockerconfigjson
        path: config.json

Following is a full example in a declarative pipeline:

pipeline {
  agent {
    kubernetes {
      label 'jnlpAgentWithGitConfig'
      yaml """
kind: Pod
metadata:
  labels:
    example: "gitconfig"
spec:
  containers:
  - name: jnlp
    workingDir: /home/jenkins/agent
    env:
    - name: HOME
      value: /home/jenkins
    volumeMounts:
      - name: jenkins-git-config-volume
        mountPath: /home/jenkins/.gitconfig
        subPath: .gitconfig
      - name: jenkins-docker-config-volume
        mountPath: /home/jenkins/.docker
  volumes:
  - name: jenkins-git-config-volume
    configMap:
      name: jenkins-git-config
      items:
      - key: .gitconfig
        path: .gitconfig
  - name: jenkins-docker-config-volume
    secret:
      secretName: my-docker-config
      items:
      - key: .dockerconfigjson
        path: config.json
"""
    }
  }
  stages {
    stage('Test Git config') {
      steps {
        sh "git config --global -l"
      }
    }
    stage('Test Docker config') {
      steps {
        sh "cat ~/.docker/config.json"
      }
    }
  }
}

Resources

Kubernetes - Volumes - EmptyDir Kubernetes - Volumes - ConfigMap Kubernetes - Volumes - Secret

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.