Cannot build images with Kaniko due to the sidecar injector

Issue

I am building an image using kaniko but it fails with an issue similar to:

Updating certificates in /etc/ssl/certs...
rm: cannot remove 'ca-certificates.crt': Device or resource busy

Environment

Explanation

Kaniko uses a very specific approach to build docker images that relies on the root file system. A particularity of the kaniko processor is that it automatically exclude directories that are mounted as volumes to the kaniko container environment. When using the sidecar injector, many volumes are automatically injected to the container pods. And this may impact kaniko’s build when images are doing operations under the mounted file systems, for example /etc/ssl/certs.

Resolution

The solution is to disable the sidecar injection for the pod by adding the annotation com.cloudbees.sidecar-injector/inject:no so that the certificates are not automatically injected

kind: Pod
metadata:
  name: kaniko
  annotations:
    com.cloudbees.sidecar-injector/inject: no
spec:
  containers:
  - name: jnlp
    workingDir: /home/jenkins
  - name: kaniko
    workingDir: /home/jenkins
    image: gcr.io/kaniko-project/executor:debug
    imagePullPolicy: Always
    command:
    - /busybox/cat
    tty: true▼

Disable injection for the kaniko container only

If the certificates are required for the other containers in the pod, the volumes may be injected manually to those containers. For example:

---
kind: Pod
metadata:
  annotations:
    com.cloudbees.sidecar-injector/inject: no
spec:
  containers:
  - name: jnlp
    volumeMounts:
      - name: ca-bundles
        mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
        subPath: ca-certificates.crt
      - name: ca-bundles
        mountPath: /etc/pki/java/cacerts
        subPath: cacerts
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    imagePullPolicy: Always
    command:
    - /busybox/cat
    tty: true
  volumes:
    - name: ca-bundles
      configMap:
        name: ca-bundles

---
kind: Pod
metadata:
  name: kaniko
  annotations:
    com.cloudbees.sidecar-injector/inject: no
spec:
  containers:
  - name: jnlp
    workingDir: /home/jenkins
    volumeMounts:
      - name: ca-bundles
        mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
        subPath: ca-certificates.crt
      - name: ca-bundles
        mountPath: /etc/pki/java/cacerts
        subPath: cacerts
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    imagePullPolicy: Always
    command:
    - /busybox/cat
    tty: true
  volumes:
    - name: ca-bundles
      configMap:
        name: ca-bundles▼

Enable custom certificates for the kaniko container

Disabling the injection for the kaniko container can be problematic if the kaniko command needs them. For instance, if the image kaniko builds needs to be pushed to a registry using a custom certificate. In this case, in addition to disabling the injection, you can also mount the certificate to a special directory (/kaniko/ssl/certs/ca-certificates.crt) for kaniko to use them.

Here is a complete example with:

the injection disabled at pod level
the certificates installed in the JNLP container
the certificates installed for kaniko in the kaniko container

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
  annotations:
    com.cloudbees.sidecar-injector/inject: no
spec:
  containers:
    - name: jnlp
      volumeMounts:
        - name: ca-bundles
          mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
          subPath: ca-certificates.crt
        - name: ca-bundles
          mountPath: /etc/pki/java/cacerts
          subPath: cacerts
    - name: kaniko
      image: gcr.io/kaniko-project/executor:debug
      command:
        - /busybox/cat
      tty: true
      volumeMounts:
        - name: ca-bundles
          mountPath: /kaniko/ssl/certs/ca-certificates.crt
          subPath: ca-certificates.crt
  volumes:
    - name: ca-bundles
      configMap:
        name: ca-bundles▼