Issue
-
I am building an image using kaniko but it fails with an issue similar to:
Updating certificates in /etc/ssl/certs... rm: cannot remove 'ca-certificates.crt': Device or resource busy
Explanation
Kaniko uses a very specific approach to build docker images that relies on the root file system. A particularity of the kaniko processor is that it automatically exclude directories that are mounted as volumes to the kaniko container environment. When using the sidecar injector, many volumes are automatically injected to the container pods. And this may impact kaniko’s build when images are doing operations under the mounted file systems, for example /etc/ssl/certs
.
Resolution
The solution is to disable the sidecar injection for the pod by adding the annotation com.cloudbees.sidecar-injector/inject:no
so that the certificates are not automatically injected
kind: Pod
metadata:
name: kaniko
annotations:
com.cloudbees.sidecar-injector/inject: no
spec:
containers:
- name: jnlp
workingDir: /home/jenkins
- name: kaniko
workingDir: /home/jenkins
image: gcr.io/kaniko-project/executor:debug
imagePullPolicy: Always
command:
- /busybox/cat
tty: true
Disable injection for the kaniko container only
If the certificates are required for the other containers in the pod, the volumes may be injected manually to those containers. For example:
---
kind: Pod
metadata:
annotations:
com.cloudbees.sidecar-injector/inject: no
spec:
containers:
- name: jnlp
volumeMounts:
- name: ca-bundles
mountPath: /etc/ssl/certs/ca-certificates.crt
subPath: ca-certificates.crt
- name: ca-bundles
mountPath: /etc/ssl/certs/java/cacerts
subPath: cacerts
- name: kaniko
image: gcr.io/kaniko-project/executor:debug
imagePullPolicy: Always
command:
- /busybox/cat
tty: true
volumes:
- name: ca-bundles
configMap:
defaultMode: 420
name: ca-bundles
---
kind: Pod
metadata:
name: kaniko
annotations:
com.cloudbees.sidecar-injector/inject: no
spec:
containers:
- name: jnlp
workingDir: /home/jenkins
volumeMounts:
- name: ca-bundles
mountPath: /etc/ssl/certs/ca-certificates.crt
subPath: ca-certificates.crt
- name: ca-bundles
mountPath: /etc/ssl/certs/java/cacerts
subPath: cacerts
- name: kaniko
image: gcr.io/kaniko-project/executor:debug
imagePullPolicy: Always
command:
- /busybox/cat
tty: true
volumes:
- name: ca-bundles
configMap:
defaultMode: 420
name: ca-bundles
Enable custom certificates for the kaniko container
Disabling the injection for the kaniko container can be problematic if the kaniko command needs them.
For instance, if the image kaniko builds needs to be pushed to a registry using a custom certificate.
In this case, in addition to disabling the injection, you can also mount the certificate to a special directory (/kaniko/ssl/certs/ca-certificates.crt
) for kaniko to use them.
Here is a complete example with:
-
the injection disabled at pod level
-
the certificates installed in the JNLP container
-
the certificates installed for kaniko in the kaniko container
apiVersion: v1
kind: Pod
metadata:
name: kaniko
annotations:
com.cloudbees.sidecar-injector/inject: no
spec:
containers:
- name: jnlp
volumeMounts:
- name: ca-bundles
mountPath: /etc/ssl/certs/ca-certificates.crt
subPath: ca-certificates.crt
- name: ca-bundles
mountPath: /etc/ssl/certs/java/cacerts
subPath: cacerts
- name: kaniko
image: gcr.io/kaniko-project/executor:debug
command:
- /busybox/cat
tty: true
volumeMounts:
- name: ca-bundles
mountPath: /kaniko/ssl/certs/ca-certificates.crt
subPath: ca-certificates.crt
volumes:
- name: ca-bundles
configMap:
defaultMode: 420
name: ca-bundles