Issue
-
The Sidecar Injector does not inject the configuration data and the Sidecar Injector pod container logs shows the following error:
TLS handshake error from <IP>:<PORT>: remote error: tls: bad certificate
Environment
-
sidecar injector v1 (manual installation scripts < 2.164.2.1)
-
sidecar injector v2 < 2.1.2
Explanation
Sidecar injector (and Admission Controllers in general) needs a keypair in order to secure its access from the Kubernetes API Server using TLS:
-
TLS key and certificate used to set up TLS for the webhook server (i.e. the sidecar injector)
-
CA Certificate / Public Key used by the
MutatingWebhookConfiguration
to validate the webhook server (i.e. the sidecar injector) certificate
The TLS key and certificate used to set up TLS for the Sidecar Injector server are generated when deploying the Sidecar Injector (from helm chart or scripts). A corresponding TLS Secret is created from the key and certificate generated and this secret is mounted to the Sidecar Injector deployment:
-
In Sidecar Injector v1, this used to be done by the script
webhook-create-signed-cert.sh
and the TLS secret name issidecar-injector-webhook-certs
by default -
In Sidecar Injector v2, this is done by a Kubernetes Job and the TLS secret name is
cloudbees-sidecar-injector
by default
The CA Certificate / Public Key used by the Kubernetes API Server to validate the Sidecar Injector server certificate is also automatically set when deploying the Sidecar Injector (from helm chart or scripts). It is added as the .webhooks[].clientConfig.caBundle
in the MutatingWebhookConfiguration
of the Sidecar Injector.
The TLS handshake error from <IP>:<PORT>: remote error: tls: bad certificate
may happens in the following scenario:
-
If Sidecar Injector has been working for a while and this just started happening, most likely the TLS certificate of the Sidecar Injector server expired.
-
If this happened when deploying the Sidecar Injector, most likely the Sidecar Injector deployment produced a bad / wrong certificate or is not able to request one.
Note the following chronology:
-
Sidecar Injector until version 2.1.0 relies on kubernetes Certificate Signing Request (CSR) that can cause problems in environment where the Kubernetes API server CA differs from the Cluster Signing Certificate:
-
The Sidecar Injector TLS key and certificate are retrieved (rather issued) by Kubernetes through a Certificate Signing Request
-
The CA Certificate / Public Key used to validate the Sidecar Injector server certificate the Kubernetes API Server CA (mounted in all pods)
-
-
Sidecar Injector v1 scripts prior to version 2.164.2.1 have a known issue in the way the CA Authority
caBundle
to use in theMutatingWebhookConfiguration
is retrieved. -
Sidecar Injector v2 starting from version 2.0.7 introduce a feature that automatically renew the Server TLS certificate before it expires.
-
Sidecar Injector v2 starting from version 2.1.0 does not rely on Certificate Signing Request anymore and creates its own self-signed certificate whose public key is injected in the
MutatingWebhookConfiguration
object so that the API server can contact it securely. -
Sidecar Injector v2 starting version 2.1.2 guarantee that the Sidecar Injector server - i.e. the pod - is restarted upon certificate renewal (that should happen a month before its expiration).
Related Issues
-
CPLT2-5479: sidecar-injector setup doesn’t work if CA is not provided in configmap - fixed in 2.164.1.2 packaged scripts
-
CPLT2-6615: Sidecar should handle CSR renewal automatically - fixed in 2.0.5
-
CPLT2-6774: Compatibility fixes for Kubernetes 1.18+ / Remove CSR Handling - fixed in 2.1.0
-
CPLT2-6981: Certificate renewal is causing errors until sidecar injector is restarted - fixed in 2.1.2
Resolution
The recommended solution is to upgrade or reinstall Sidecar Injector deployment to version 2.1.2 or later:
-
If running Sidecar Injector version 1.x, delete the Sidecar Injector with
kubectl delete -f sidecar-injector.yaml -n <namespace>
and install Sidecar Injector 2.1.2 or later following Using self-signed certificates in CloudBees CI on Kubernetes. -
If running Sidecar Injector version 2.x, upgrade the Sidecar Injector to version 2.1.2 or later
If upgrading from version 2.0.7 or later, the CronJob that manages certificates will not be automatically started upon upgrade. In that case, start a new Job based on it with a command like kubectl create job --from=cronjob/cloudbees-sidecar-injector -n sidecar-injector .
|
Troubleshooting
To troubleshoot such issues, check the Sidecar Injector TLS certificate (for the TLS secret) and the .clientConfig.caBundle
of the MutatingWebhookConfiguration
.
Issue with Certificate Expiration
This issue may appear if the certificate used to set up TLS for the Sidecar Injector server have expired. To confirm this, check the expiry date of this certificate:
-
In Sidecar Injector v1: check the
cert.pem
in secretsidecar-injector-webhook-certs
:kubectl get secret sidecar-injector-webhook-certs -o "go-template={{ index .data \"cert.pem\" }}" | base64 --decode | openssl x509 -enddate -noout
-
In Sidecar Injector v2: check the
tls.crt
in secretcloudbees-sidecar-injector
:kubectl get secret cloudbees-sidecar-injector -o "go-template={{ index .data \"tls.crt\" }}" | base64 --decode | openssl x509 -enddate -noout
Issue with Deployment
This can happen if the Kubernetes API Server certificate is different from the Cluster Signing Certificate. To confirm this, check the issuer and CN of the Sidecar Injector TLS certificate and the MutatingWebhookConfiguration
Public Key:
-
In Sidecar Injector v1: check the
cert.pem
in secretsidecar-injector-webhook-certs
:kubectl get secret sidecar-injector-webhook-certs -o "go-template={{ index .data \"cert.pem\" }}" -n sidecar-injector | base64 --decode | openssl -noout -subject -issuer kubectl get mutatingWebhookConfiguration sidecar-injector-webhook-cfg -o "go-template={{ (index .webhooks 0).clientConfig.caBundle }}" -n sidecar-injector | base64 --decode | openssl -noout -subject -issuer
-
In Sidecar Injector v2: check the
tls.crt
in secretcloudbees-sidecar-injector
:kubectl get secret cloudbees-sidecar-injector -o "go-template={{ index .data \"tls.crt\" }}" -n $SIDECAR_NAMESPACE | base64 --decode | openssl x509 -noout -subject -issuer kubectl get mutatingWebhookConfiguration cloudbees-sidecar-injector -o "go-template={{ (index .webhooks 0).clientConfig.caBundle }}" -n sidecar-injector | base64 --decode | openssl -noout -subject -issuer