Pod annotations disappear when using sidecar-injector

Issue

I have added annotations to controller / CJOC / Pod templates but the annotations are not reflected, only com.cloudbees.sidecar-injector/status: injected is present

Environment

Explanation

This is caused by the mutation done by the sidecar-injector in version lower than 1.2.1 that does not preserve the existing annotations of the pod on which the mutation happens.

Resolution

The solution is to upgrade the sidecar-injector deployment image to version 1.2.1 or later.

Solution 1: Update the configuration

Edit the configuration file of the sidecar-injector deployment - for example when following the manual installation of the sidecar-injector the file is sidecar-injector-ca-bundle.yaml - and change the container image of the sidecar-injector-webhook-deployment deployment (replace NEW_VERSION by the version to upgrade to):

[...]
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sidecar-injector-webhook-deployment
  namespace: sidecar-injector
  labels:
    app: sidecar-injector
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: sidecar-injector
    spec:
      containers:
        - name: sidecar-injector
          image: cloudbees/sidecar-injector:NEW_VERSION
[...]

Then apply the changes with:

kubectl apply -f sidecar-injector-ca-bundle.yaml

Solution 2: Patch the deployment

Another solution is to patch the sidecar injector deployment directly (replace NEW_VERSION by the version to upgrade to):

kubectl patch deployment sidecar-injector-webhook-deployment \
    -p '{"spec":{"template":{"spec":{"containers":[{"name":"sidecar-injector","image":"cloudbees/sidecar-injector:NEW_VERSION"}]}}}} \
    -n sidecar-injector'

We strongly recommend keeping the configuration of kubernetes resources in version controlled files.

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.