Required Data: Sidecar Injector

Issue

I have deployed the sidecar-injector but certificates don’t seem to be injected

Quick check

Check the description of the pod impacted (replace $POD_NAME with the name of the pod to check the injection for, for example cjoc-0):

kubectl get pod $POD_NAME -o yaml▼

and check the volumes and volumeMounts to validate that the expected volumes are injected. You should see something like the following:

    volumeMounts:
    - mountPath: /etc/ssl/certs/ca-certificates.crt
      name: ca-bundles
      subPath: ca-certificates.crt
    [...]
  volumes:
  - configMap:
      defaultMode: 420
      name: ca-bundles
    name: ca-bundles
    [...]

If that is the case, then it can be assumed that injection is working as expected and what needs troubleshooting is the SSL. Please follow the guide SSL Certificates Troubleshooting.

Required Data for sidecar-injector

This article describes how to collect the minimum required information for sidecar-injector on a CloudBees Core on Core Modern installation so that it can be efficiently troubleshooted.

If the required data is bigger than 50 MB you will not be able to use ZenDesk to upload all the information. In this case, we would like to encourage you to use our upload service in order to attach all the required information.

Environment

Automatic Data collection

This is the preferred method if you are using a product supported by the cbsupport CLI.

Current products supporting collecting sidecar-injector data are:

CloudBees CI (CloudBees Core) on Modern Cloud Platforms

Steps to follow are:

Install and configure cbsupport following Using cbsupport CLI to collect the requested data
Run cbsupport required-data sidecar-injector
In the command prompt, select the namespace where the sidecar injector has been deployed, commonly sidecar-injector
Collect the archive generated in the working directory of cbsupport and attach it to the ticket using our upload service

Manual Data collection

Required Data check list

Kubernetes sidecar-injector resources details
Kubernetes Namespace labelled for injection
Kubernetes sidecar-injector pod logs
Kubernetes sidecar-injector job pod logs
Kubernetes CloudBees Core resources details
Kubernetes Impacted pod injection logs
Kubernetes Sidecar Injector Server Certificate
Kubernetes Sidecar Injector CSR (< 2.1.0 only)
Support bundle from impacted Instance

Pre-Requisites

To facilitate the retrieval of data, export the following variables:

CB_NAMESPACE=<cloudbees-core-namspace>
SIDECAR_NAMESPACE=<sidecar-namespace>

Replace:

<cloudbees-core-namespace> by the namespace where CloudBees Core is deployed
<sidecar-namespace> by the namespace where the sidecar-injector is deployed, usually sidecar-injector

Kubernetes `sidecar-injector` resources details

Resources of the sidecar-injector deployment:

kubectl get deployment,cm,pod,svc,ep,mutatingWebhookConfigurations,job,cronjob -n $SIDECAR_NAMESPACE -o yaml > sidecar-injector-details.yaml
kubectl get deployment,cm,pod,svc,ep,mutatingWebhookConfigurations,job,cronjob -n $SIDECAR_NAMESPACE -o wide > sidecar-injector-details.txt▼

Kubernetes Namespace labelled for injection

The list of namespaces labelled for injection:

kubectl get namespaces -L sidecar-injector > sidecar-injector-labels.txt▼

Kubernetes `sidecar-injector` pod logs

Get the logs of the sidecar-injector pod:

kubectl logs -n $SIDECAR_NAMESPACE $(kubectl get pod -n $SIDECAR_NAMESPACE -o jsonpath='{.items[0].metadata.name}' -l app.kubernetes.io/name=cloudbees-sidecar-injector) > sidecar-injector-pod.log▼

For Sidecar Injector 1.x - i.e. non-helm installation:

kubectl logs -n $SIDECAR_NAMESPACE $(kubectl get pod -n $SIDECAR_NAMESPACE -o jsonpath='{.items[0].metadata.name}' -l app=sidecar-injector) > sidecar-injector-pod.log▼

Kubernetes `sidecar-injector` job pods logs

Get the logs of the sidecar-injector pods managed scheduled by the init job and/or the cronjob:

for pod_name in `kubectl get pod -n $SIDECAR_NAMESPACE -o jsonpath='{.items[*].metadata.name}' -l job-name`; do kubectl logs $pod_name -n $SIDECAR_NAMESPACE > $pod_name.log; done▼

Kubernetes CloudBees Core resources details

Details of the CloudBees Core resources:

kubectl get node,sts,pod,svc,ing,ep,cm,pvc,pv -o yaml -n $CB_NAMESPACE > cje2-details.yaml
kubectl get node,sts,pod,svc,ing,ep,cm,pvc,pv -o wide -n $CB_NAMESPACE > cje2-details.txt▼

Kubernetes Impacted pod injection logs

Check the injection inside the pod. For example for Debian / Alpine / Ubuntu, certificates are injected at /etc/ssl/certs/ and /etc/ssl/certs/java/ (replace $POD_NAME with the name of the pod to check the injection for, for example cjoc-0):

kubectl exec -ti $POD_NAME -n $CB_NAMESPACE -- ls -lR /etc/ssl/certs/ > pod-injection.log▼

Kubernetes Sidecar Injector Server Certificate

Get the certificate of the Sidecar Injector server from the secret:

kubectl get secret cloudbees-sidecar-injector -o 'go-template={{index .data "tls.crt"}}' -n $SIDECAR_NAMESPACE > tls.crt.base64▼

For Sidecar Injector 1.x - i.e. non-helm installation but the version can also be found with kubectl get pod $POD_NAME -n $SIDECAR_NAMESPACE -o jsonpath='{.spec.containers[0].image}':

kubectl get secret sidecar-injector-webhook-certs -o 'go-template={{index .data "cert.pem"}}' -n $SIDECAR_NAMESPACE > cert.pem.base64▼

Kubernetes Sidecar Injector CSR logs (version < 2.1.0 only)

Get the CSR generated by the sidecar injector if it still exists:

kubectl get csr cloudbees-sidecar-injector -o yaml > sidecar-injector-csr.yaml
kubectl get csr cloudbees-sidecar-injector -o wide > sidecar-injector-csr.txt▼