Sidecar injector for self-signed certificates on OpenShift

5 minute read

CloudBees CI includes an option called sidecar injector. This option lets you use a self-signed certificate or a custom certificate authority (CA) to access internal HTTPS services, such as an SCM repository or an artifact repository.

Sidecar injector is designed only to trust services that are secured with custom or self-signed certificates. It is not intended to be used to secure a CloudBees CI cluster using HTTPS and should not be set up for that purpose.

Using sidecar injector provides the following benefits:

  • Satisfies the enterprise security policies for organizations that use custom certificate authorities.

  • Removes the cost of purchasing commercially signed certificates for internal websites.

  • Saves time because organizations don’t have to wait for signed certificates.

To use sidecar injector, create a certificate bundle that overwrites the default certificate bundle. Then, set up sidecar injector to inject that certificate bundle into all containers of all scheduled Kubernetes pods in a labeled namespace.

Prerequisites for using sidecar injector on OpenShift

The following items are required:

  • A supported version of OpenShift. See Supported platforms for CloudBees CI on modern cloud platforms - Kubernetes reference platforms.

  • The MutatingAdmissionWebhook admission controller is enabled for your cluster.

    To verify the admission controller is enabled for your cluster, enter the following command:

    $ oc api-versions | grep admissionregistration.k8s.io/v1beta1

    The result should be:

    admissionregistration.k8s.io/v1beta1
  • For OpenShift 3.x only, add the admission controllers to the master-config.yaml file as follows:

    admissionConfig: pluginConfig: ValidatingAdmissionWebhook: configuration: kind: DefaultAdmissionConfig apiVersion: v1 disable: false MutatingAdmissionWebhook: configuration: kind: DefaultAdmissionConfig apiVersion: v1 disable: false

Network requirements for sidecar injector

The sidecar injector listens to HTTPS requests on port 443, and the firewall rules of that port must be configured accordingly:

From To Port Description

Kubernetes Control Plane

Kubernetes Node(s)

443

Allow Kubernetes control plane to communicate with sidecar-injector pod(s)

Kubernetes Node(s)

Kubernetes Control Plane

443

Allow incoming requests from sidecar-injector pod(s)

In environments where aggregator routing is enabled, the API Server routes the webhook requests directly to the Sidecar Injector Endpoint that listens on port 8443, rather than the Kubernetes Service that listens on port 443. In such environments, the firewall rules must allow the Kubernetes control plane to communicate with Kubernetes node(s) on the port 8443.

Installing self-signed certificates on OpenShift

This procedure requires a context with the cluster-admin privilege to create the MutatingWebhookConfiguration.

Follow all of the following instructions in the order listed to ensure that sidecar injector works correctly with your self-signed certificates:

Creating a certificate bundle on OpenShift

These instructions assume that you are working in the project where CloudBees CI is installed, and that the certificate you want to install is named mycertificate.pem.

If you are using a self-signed certificate, add the certificate itself. If you are using a certificate issued by a custom root CA, add the root CA itself.

  1. Copy reference files locally:

    On CloudBees CI 2.204.1.3 or newer:

    $ oc cp cjoc-0:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ./ca-certificates.crt $ oc cp cjoc-0:/etc/pki/ca-trust/extracted/java/cacerts ./cacerts

    On earlier versions, or when overriding the Docker image selection to use the Alpine base:

    $ oc cp cjoc-0:/etc/ssl/certs/ca-certificates.crt ./ca-certificates.crt $ oc cp cjoc-0:/etc/ssl/certs/java/cacerts ./cacerts

    Alternatively, in case you do not have an operations center pod running, you can retrieve the files from the image without deploying in OpenShift.

    For this you will need to have a working Docker installation.

    You can then run:

    CBCI_VERSION="2.249.2.4" CONTAINER_ID=$(docker create cloudbees/cloudbees-cloud-core-oc:${CBCI_VERSION}) # adapt for each file you need to retrieve docker cp ${CONTAINER_ID}:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem . docker rm ${CONTAINER_ID}
  2. Add the root CA to the system certificate bundle:

    $ cat mycertificate.pem >> ca-certificates.crt
  3. Add the root CA to the Java cacerts:

    $ keytool -import -noprompt -keystore cacerts -file mycertificate.pem -storepass changeit -alias service-mycertificate;
    Make sure that mycertificate.pem contains only one certificate. keytool does not support importing multiple certificates from a single file.
  4. Use the two files above to create a configmap for each labeled namespace that uses sidecar injector:

    $ oc create configmap --from-file=ca-certificates.crt,cacerts ca-bundles

Setting up the sidecar injector on OpenShift using Helm install

This procedure shows you how to set up sidecar injector using the helm install method. If you are unable to use helm install, go to Setting up sidecar injector on OpenShift using Helm template.

  1. Create a project to deploy the sidecar injector:

    $ oc new-project cloudbees-sidecar-injector
  2. Type the following command:

    helm install cloudbees-sidecar-injector cloudbees/cloudbees-sidecar-injector --namespace cloudbees-sidecar-injector
  3. Verify that the sidecar-inject-webhook pod is running:

    $ oc --namespace cloudbees-sidecar-injector get pods NAME READY STATUS RESTARTS AGE sidecar-injector-webhook-deployment-bbb689d69-882dd 1/1 Running 0 5m $ oc --namespace cloudbees-sidecar-injector get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE sidecar-injector-webhook-deployment 1 1 1 1 5m

Setting up the sidecar injector on OpenShift using Helm template

Use this procedure to set up sidecar injector if you are unable to use helm install. This procedure uses the helm template method.

  1. Create a project to deploy the sidecar injector:

    $ oc new-project cloudbees-sidecar-injector
  2. Enter the following command:

    helm template cloudbees-sidecar-injector cloudbees/cloudbees-sidecar-injector --namespace cloudbees-sidecar-injector --api-versions "route.openshift.io/v1" > sidecar-injector.yaml
  3. Enter the following command:

    oc apply --namespace cloudbees-sidecar-injector -f sidecar-injector.yaml.
  4. Verify that the sidecar-inject-webhook pod is running:

    $ oc --namespace cloudbees-sidecar-injector get pods NAME READY STATUS RESTARTS AGE sidecar-injector-webhook-deployment-bbb689d69-882dd 1/1 Running 0 5m $ oc --namespace cloudbees-sidecar-injector get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE sidecar-injector-webhook-deployment 1 1 1 1 5m

Configuring a namespace for sidecar injector on OpenShift

  1. Label the namespace where CloudBees CI is installed with sidecar-injector=enabled:

    In the command below, replace "mynamespace" with the name of your namespace. For example, if your namespace is "cb", use the command kubectl label namespace cb sidecar-injector=enabled. This causes pods that start in that namespace to have the sidecar injected.
    $ oc label namespace mynamespace sidecar-injector=enabled
  2. Check the configuration:

    $ oc get namespace -L sidecar-injector NAME STATUS AGE SIDECAR-INJECTOR default Active 18h mynamespace Active 18h enabled kube-public Active 18h kube-system Active 18h

Verifying the namespace for sidecar injector on OpenShift

  1. Switch to the CloudBees CI project:

    $ oc project mynamespace
  2. Deploy an app in the OpenShift cluster. For example, the sleep app:

    # cat <<EOF | oc create -f - apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sleep spec: replicas: 1 template: metadata: labels: app: sleep spec: containers: - name: sleep image: busybox command: ["/bin/sleep","infinity"] EOF
  3. Verify that injection has occurred:

    $ oc get pods -o 'go-template={{range .items}}{{.metadata.name}}{{"\n"}}{{range $key,$value := .metadata.annotations}}* {{$key}}: {{$value}}{{"\n"}}{{end}}{{"\n"}}{{end}}' sleep-d5bf9d8c9-bfglq * com.cloudbees.sidecar-injector/status: injected

Applying the certificate bundle

After you have completed the setup for sidecar injector, you can use your custom CA across your cluster.

To apply the new certificate bundle:

  1. Use CJOC_RUL/restart to restart operations center and the pod.

  2. Restart any running managed controllers.

When new build agents are scheduled, the certificate bundle is automatically applied and permits connection to remote endpoints using your certificates.

Advanced configuration for sidecar injector on OpenShift

Disabling Injection on a specific pod

To disable implicit injection for a specific pod, annotate it with com.cloudbees.sidecar-injector/inject: no

Making injection explicit

By default, injection is implicit and applies to all pods created in the labeled namespace(s). However, you can alternately enable injection on only the pod(s) that explicitly require it.

To make injection explicit for a given pod:

  1. Edit the sidecar-injector-webhook-configmap configmap and specify requiresExplicitInjection: true.

  2. To enable injection on a specific pod, annotate the pod with com.cloudbees.sidecar-injector/inject: yes.

Troubleshooting self-signed certificates on OpenShift

The sleep pod can’t be created

There is an error on pod creation, such as "certificate signed by unknown authority", or the sidecar-injector logs contain the following: http: TLS handshake error from aaa.bbb.ccc.ddd:nnnnn: remote error: tls: bad certificate.

This can happen if the API server TLS certificate differs from the cluster signing certificate. To fix this, you need to provide the cluster signing certificate as an input to the installation.

Create a values.yaml file as follows. Replace the content with your own certificate.

caBundleCrt: |- -----BEGIN CERTIFICATE----- MIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl cm5ldGVzMB4XDTE5MDcyOTA3MzQyMloXDTI5MDcyNjA3MzQyMlowFTETMBEGA1UE AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALgQ fjIm8iVEpmX3tlcUzuH2BAQGzK0utC6S0hhqnK6zlQ9iGDAwDpAiwGB9VzcJSfK7 fr3rx4zT9rWfVeot+ARQV/NPxSUpPlGK8WsRleg5wKyUdnE1xKZDly2l2Vpqlr0J GnMXb2A0roi685XZo6iQALLfo+rtWQ2y2JLXzGYYCB1sAUX3hM3qbYmIMReBIyMX YGUUdaMuWU1YazKy3eJ84Am7l9ZXlMm7infJlAFsM3BCKed9ZxO2KxTvhWv1qbUk Bj3GJrL2bJfQi3B6h0piiBDt6YeI3U8yU4EyxtMKQwQXs9T1zHloc6RmVGYYVAEl HquW/XIk4ebWTxYND6UCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB /wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBABsJvnyo14a8FR6y4JGSX1SXSmtS uRiWH6qo5Ou+4zIw0LIhDQTtaN44G86BzhvQFdeQzfvyrKXfuYUTOQz/iYPNFsO6 FOcDg9EcA19n5tSGp8SniyDEe6EhBWa5A9UR2RPEg/8NZRoeZ/2G9SjUqoa/Erxn IPlZvNu+gMEK7etysUQ33s4fp+jD6p0pbWKgSQAiDRVHi3Khlhcn7DfM0ncrSQBs vgFPSEczjpl8LR6c0pLSdPUHgdK6pDLTYdtdytRNfJAVjAREYL4uSb8I5NKdkgEz BLvnDfdDoCeOZoaeLR68jCltNfdzT5/d1v086i7uRepGhQ5w7ehtPuZ+0U8= -----END CERTIFICATE-----
If running on AWS EKS, you can get the API server TLS certificate by opening the AWS Console, navigating to menu:EKS[Clusters > Your Cluster > Overview, and copying the Certificate authority. Then, run echo 'COPIED_VALUE' | base64 -d to get the certificate. It is also found at /etc/kubernetes/pki/ca.crt from any Kubernetes node.

Then, run the installation again.

helm delete cloudbees-sidecar-injector -n cloudbees-sidecar-injector kubectl delete MutatingWebhookConfiguration cloudbees-sidecar-injector kubectl delete namespace cloudbees-sidecar-injector kubectl create namespace cloudbees-sidecar-injector helm install cloudbees-sidecar-injector cloudbees/cloudbees-sidecar-injector --namespace cloudbees-sidecar-injector --values values.yaml

The sleep pod is created but stays in ContainerCreating state

Describe the pod:

kubectl describe po sleep

You may get an error like the following:

Warning FailedMount 2s (x3 over 3s) kubelet, docker-desktop MountVolume.SetUp failed for volume "bundles" : configmap "ca-bundles" not found

If so, please verify that you created the certificate bundle mentioned above. Also check that you are working in the expected namespace. A configmap called ca-bundles must be created in each namespace you wish to use sidecar injector with.

Remove a previous broken installation and perform a fresh installation

If you have a previous installation that was not installed correctly, or is failing to upgrade, you can follow these steps to remove the previous installation and perform a fresh installation.

  1. Retrieve the existing Helm values:

    helm get values -n cloudbees-sidecar-injector cloudbees-sidecar-injector > cloudbees-sidecar-injector-values.yaml
  2. Edit the generated cloudbees-sidecar-injector-values.yaml file and remove USER-SUPPLIED VALUES: at the top of the file. If the file only has null remaining, then the previous installation did not use any values; omit -f cloudbees-sidecar-injector-values.yaml when running the new installation.

  3. Uninstall the cloudbees-sidecar-injector release, delete the MutatingWebhookConfiguration, and delete the namespace:

    helm uninstall -n cloudbees-sidecar-injector cloudbees-sidecar-injector kubectl delete MutatingWebhookConfiguration cloudbees-sidecar-injector kubectl delete namespace cloudbees-sidecar-injector
  4. Create the namespace, and install cloudbees-sidecar-injector:

    kubectl create namespace cloudbees-sidecar-injector helm repo update helm install -n cloudbees-sidecar-injector cloudbees-sidecar-injector cloudbees/cloudbees-sidecar-injector -f cloudbees-sidecar-injector-values.yaml
In August 2020, the Jenkins project voted to replace the term master with controller. We have taken a pragmatic approach to cleaning these up, ensuring the least amount of downstream impact as possible. CloudBees is committed to ensuring a culture and environment of inclusiveness and acceptance - this includes ensuring the changes are not just cosmetic ones, but pervasive. As this change happens, please note that the term master has been replaced through the latest versions of the CloudBees documentation with controller (as in managed controller, client controller, team controller) except when still used in the UI or in code.