Pod annotations disappear when using sidecar-injector

Issue

I have added annotations to controller / CJOC / Pod templates but the annotations are not reflected, only com.cloudbees.sidecar-injector/status: injected is present

Environment

Explanation

This is caused by the mutation done by the sidecar-injector in version lower than 1.2.1 that does not preserve the existing annotations of the pod on which the mutation happens.

Resolution

The solution is to upgrade the sidecar-injector deployment image to version 1.2.1 or later.

Solution 1: Update the configuration

Edit the configuration file of the sidecar-injector deployment - for example when following the manual installation of the sidecar-injector the file is sidecar-injector-ca-bundle.yaml - and change the container image of the sidecar-injector-webhook-deployment deployment (replace NEW_VERSION by the version to upgrade to):

[...]
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sidecar-injector-webhook-deployment
  namespace: sidecar-injector
  labels:
    app: sidecar-injector
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: sidecar-injector
    spec:
      containers:
        - name: sidecar-injector
          image: cloudbees/sidecar-injector:NEW_VERSION
[...]

Then apply the changes with:

kubectl apply -f sidecar-injector-ca-bundle.yaml

Solution 2: Patch the deployment

Another solution is to patch the sidecar injector deployment directly (replace NEW_VERSION by the version to upgrade to):

kubectl patch deployment sidecar-injector-webhook-deployment \
    -p '{"spec":{"template":{"spec":{"containers":[{"name":"sidecar-injector","image":"cloudbees/sidecar-injector:NEW_VERSION"}]}}}} \
    -n sidecar-injector'

We strongly recommend keeping the configuration of kubernetes resources in version controlled files.