Article ID:360001566111
7 minute readKnowledge base
  • I need to use Docker Multi-stage builds on my build agents based on Docker Agent Templates.

  • I need to use a different Docker version than the one provided on my build agents based on Docker Agent Templates.

  • I need to run Docker commands with a local bind mount (-v $(pwd):/app) on my build agents based on Docker Agent Templates.

  • I do not want "dangling" containers to last after my builds terminate.

  • I do not want other builds to see or interact with my containers ("re-entrant build").

  • My build, running on an Agent Template, cannot use Docker with the following error:

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock:
Get https://%2Fvar%2Frun%2Fdocker.sock/v1.35/containers/json:
dial unix /var/run/docker.sock: connect: permission denied


Those issues are caused by the default Docker Agent Template of CJEv1 configuration, referred to as "Docker on Docker (DonD)", "sibling containers", or "docker.sock socket sharing".

This default "docker" template works by sharing the file /var/run/docker.sock and the workspace inside your build agent container, to allow communication from your build to the worker machine’s Docker Engine. It allows Docker layer caching by default which improves build time.

Docker On Docker Diagram

To solve your issue, you need to configure a new Docker Agent Template providing a self-service and ephemeral Docker Engine, which your build will use instead of the worker machine’s Docker Engine.

This pattern is referred as "Docker in Docker", or "nested containerization".

Docker in Docker diagram

This article covers the following steps to achieve this:

  1. Prepare a custom Docker Image

  2. Configure a new Docker Agent Template

  3. Validate the setup

  4. Configure the caching of Docker images

  5. Discussion and implications of this solution

1. Prepare a custom Docker Image

The solution provided in this article assumes that you are familiar with building a custom Docker Image and hosting it on a Docker registry. If you are not, start with and

You need to create an image with the following elements:

  • Dockerfile:

# Official Docker Image from
# Set the Docker version you want to use
FROM docker:18.02-dind

# Defining default variables and build arguments
ARG user=jenkins
ARG group=jenkins
ARG uid=1000
ARG gid=1000
ARG jenkins_user_home=/home/${user}

ENV JENKINS_USER_HOME=${jenkins_user_home} \
  LANG=C.UTF-8 \
  JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk \
  PATH=${PATH}:/usr/local/bin:/usr/lib/jvm/java-1.8-openjdk/jre/bin:/usr/lib/jvm/java-1.8-openjdk/bin \
  DOCKER_IMAGE_CACHE_DIR=/docker-cache \

# Install required packages for running a Jenkins agent
RUN apk add --no-cache \
  bash \
  curl \
  ca-certificates \
  git \
  openjdk8 \
  unzip \
  tar \

# Set up default user for jenkins
RUN addgroup -g ${gid} ${group} \
  && adduser \
    -h "${jenkins_user_home}" \
    -u "${uid}" \
    -G "${group}" \
    -s /bin/bash \
    -D "${user}" \
  && echo "${user}:${user}" | chpasswd

# Adding the default user to groups used by Docker engine
# "docker" for avoiding sudo, and "dockremap" if you enable user namespacing
RUN addgroup docker \
  && addgroup ${user} docker \
  && addgroup ${user} dockremap

# Custom start script
COPY ./entrypoint.bash /usr/local/bin/entrypoint.bash

# Those folders should not be on the Docker "layers"
VOLUME ${jenkins_user_home} /docker-cache /tmp

# Default working directory
WORKDIR ${jenkins_user_home}

# Define the "default" entrypoint command executed on the container as PID 1
ENTRYPOINT ["/sbin/tini","-g","--","bash","/usr/local/bin/entrypoint.bash"]
# This entrypoint is dedicated to start Docker Engine (in Docker)
# For CloudBees Jenkins Enterprise 1.x Agent Templates
# Cf.

set -e

if [ $# -gt 0 ]
  ## Default Docker flag. It can be overwritten by an environment variable

    # Default settings: we let the script autodetect the "best"
    # storage driver for Docker based on the host kernel features
    if [ -f "/proc/filesystems" ]
      ## Best case: we can use overlayFS 2 (overlay2 for Docker)
      if [ "$(grep overlay /proc/filesystems)" ]
      elif [ "$(grep aufs /proc/filesystems)" ]
      # overlay not available: let's try aufs, the Docker "original" layer FS
        # Other drivers exists: btrfs, zfs, but they need advanced configuration
        # Check
        # We fallback to the worst case, "vfs": slow but works everywhere
      DOCKER_OPTS+=" --storage-driver ${STORAGE_DRIVER}"

  echo "== Starting Docker Engine with the following options: ${DOCKER_OPTS}"
  /usr/local/bin/ ${DOCKER_OPTS} >/docker.log 2>&1 &

  # Wait for Docker to start by checking the TCP socket locally
  ## Wait 1 second to let all process and file handle being created
  echo "== Waiting for Docker Engine process to start"
  sleep 1

  ## Try reaching the unix socket 6 times, waiting 5s between tries
  curl -XGET -sS -o /dev/null --fail \
    --retry 6 --retry-delay 5 \
    --unix-socket /var/run/docker.sock \
    https://DOCKER/_ping || (cat /docker.log && exit 1)

  ## Last check: the _ping endpoint should send "OK" on stdout
  [ "$(curl -sS -X GET --unix-socket /var/run/docker.sock http:/images/_ping)" == "OK" ]

  echo "== Docker Engine started and ready"

  # Load any "tar-gzipped" Docker image from the local cache
  if [ -n "${DOCKER_IMAGE_CACHE_DIR}" ] && [ -d "${DOCKER_IMAGE_CACHE_DIR}" ]
    echo "== Variable 'DOCKER_IMAGE_CACHE_DIR' found and pointing to an existing Directory"
    echo "== Loading following .tar files in Docker:"
    find "${DOCKER_IMAGE_CACHE_DIR}" -type f -name "*.gz" -print \
      -exec sh -c 'gunzip --stdout "$1" | docker load' -- {} \;

  # second argument is the java command line generated by CJE (passed as a single arg)
  echo "== Launching the following user-provided command: ${*}"
  exec /bin/sh -c "$@"

For the next steps, we assume that this image is built and hosted with the name

2. Configure a new Docker Agent Template

Create a new Agent Template, either in Operation Center or in one of your Managed controllers.

Configure this template with the following settings; if you are unsure about a setting, refer to the settings in the default docker Agent Template in your Operations Center:

  • Additional labels: docker-in-docker is a good start. You can add docker if you want to make it the default, but take care of disabling the default one. You can also use the Docker version if you need strict compliance: docker-18.02 in our example.

  • Resources: CPU shares and Memory needs to be adapted to your needs. Use the same values as the default Docker Agent Template to start.

  • Java Options: same as for Resources, use the default Docker Agent Template or tune it to your needs.

  • Filesystem: "Remote root directory": /jenkins

  • Definition: "Image" must be set up to your Docker image name. It would be in our example.

  • Options:

    • Remove ALL existing options unless you know what you are doing.

    • The path /var/run/docker.sock must not be shared.

    • Add a "Launch in privileged mode" option

    • Add a "Use a custom Docker shell command" option, and set the field to DUMMY

    • Add a "Parameter", and set the "Key" to rm and the "Value" to true

    • Add a "Volume" option, set the host path to a folder on the VM that will contain the cache, let’s say /DATA/docker-cache, and set "Container Path" to /docker-cache

Configuration of the new Template Agent

3. Validate the setup

You can validate that this setup works by executing the following Pipeline; the build Output should show you the full Docker Engine configuration. Verify that the version is 18.02 (or the version on your Dockerfile’s FROM instruction):

pipeline {
  agent {
      label 'docker-in-docker'
    stage('Validate Docker') {
      steps {
        sh 'docker info'

4. Configure the caching of Docker images

There is a big downside on this configuration: a fresh Docker engine is spawned on each build. It means that the "Docker cached layers" are not reusable across your builds.

However, this setup provides a facility, by preloading any file with the extension tar.gz, stored on the /docker-cache directory of the agent container.

In our example, we are mapping this folder to the VM folder /DATA_docker-cache: persistence is done at agent level; as it was the case before.

Here is a Pipeline example that will ask for a Docker image, download it, and store it into the cache:

pipeline {
  agent {
      label 'docker-in-docker'
  parameters {
    string(name: 'DOCKER_IMAGE_TO_CACHE', defaultValue: 'alpine', description: 'Which Docker Image do you want to cache?')
    stage('Validate Docker') {
      steps {
        sh '''
        ARCHIVE_NAME="$(echo "${DOCKER_IMAGE_TO_CACHE}" | sed -e 's#/#-#g' -e 's#:#_#g').tar.gz"
        ls -l "${CACHE_DIR}"
        docker pull "${DOCKER_IMAGE_TO_CACHE}"
        docker save "${DOCKER_IMAGE_TO_CACHE}" | gzip > "${CACHE_DIR}/${ARCHIVE_NAME}"

Please note we are using "gzipped" images to avoid wasting too much disk space. It can have an impact on the "agent startup time". If you have plenty of hard drive space, you can remove the "gzip/gunzip" steps and only store cache as "tar" archive.

5. Discussion and implications of this solution

You might have a read on this awesome blog article:

This article underlines 2 elements, when using "DinD" (Docker in Docker):

  • "The bad": we are launching the agent containers with the flag --privileged. It means that this container can do anything on the worker machine. But the usage of "DonD" has the same implication: Docker is run as root and can access anything on the host. So changing from "DonD" to "Dind" did not changed anything on this topic. A good read on this: to go deep dive as well.

  • "The ugly / the worse": The file storage issue are related to what happened back in 2015. Recent Docker Storage Drivers implementations (AUFS, overlayfs and devicemapper) do not have these performances or corruption issues.

The entrypoint.bash script takes care to auto-detect the best storage option at container startup:

  • The driver overlay2 is selected by default if the worker kernel supports it.

  • Otherwise, the driver aufs is selected if supported by the worker kernel.

  • If neither overlay nor aufs are supported by your kernel, then the vfs driver is used as fallback.

You can find more information on these pages:

6. Configure Access to your private docker registry (Optional)

If you are using a private docker registry, you will need to include a couple of extra configuration elements in the template.

  • You will need to add the user configuration, which will set the username or UID to use.

  • You will need to provide the data to access the docker registry as explained here.

Once that you have completed these steps you will be able to use the template in your builds.

6. Troubleshooting

It has had some cases where the container running inside the DinD image is not able to connect some URLs obtaining timeout and/or dns errors, however it works when the container runs in the build worker.

To confirm it, it’s needed to access into the container running the DinD image and execute a curlcommand to the target URL, the output should be similar to

curl -ILv https://example

* Rebuilt URL to: https://example
* Could not resolve host: example
* Closing connection 0
curl: (6) Could not resolve host: example

To workaround the issue, please add the parameter network with value host in the Docker Agent template configured. This way, the docker container will use the network configured in the build worker instead of the docker bridge

Tested product/plugin versions

The latest update of this article has been tested with: