Sidecar injector for self-signed certificates on OpenShift

CloudBees CI includes an option called sidecar injector. This option lets you use a self-signed certificate or a custom certificate authority (CA) to access internal HTTPS services, such as an SCM repository or an artifact repository.

Sidecar injector is designed only to trust services that are secured with custom or self-signed certificates. It is not intended to be used to secure a CloudBees CI cluster using HTTPS and should not be set up for that purpose.

Using sidecar injector provides the following benefits:

  • Satisfies the enterprise security policies for organizations that use custom certificate authorities.

  • Removes the cost of purchasing commercially signed certificates for internal websites.

  • Saves time because organizations don’t have to wait for signed certificates.

To use sidecar injector, create a certificate bundle that overwrites the default certificate bundle. Then, set up sidecar injector to inject that certificate bundle into all containers of all scheduled Kubernetes pods in a labeled namespace.

Prerequisites for using sidecar injector on OpenShift

The following items are required:

  • A supported version of OpenShift. See Supported platforms for CloudBees CI on modern cloud platforms - Kubernetes reference platforms.

  • The MutatingAdmissionWebhook admission controller is enabled for your cluster.

    To verify the admission controller is enabled for your cluster, enter the following command:

    $ oc api-versions | grep admissionregistration.k8s.io/v1beta1

    The result should be:

    admissionregistration.k8s.io/v1beta1
  • For OpenShift 3.x only, add the admission controllers to the master-config.yaml file as follows:

    admissionConfig:
      pluginConfig:
        ValidatingAdmissionWebhook:
          configuration:
            kind: DefaultAdmissionConfig
            apiVersion: v1
            disable: false
        MutatingAdmissionWebhook:
          configuration:
            kind: DefaultAdmissionConfig
            apiVersion: v1
            disable: false

Network requirements for sidecar injector

The sidecar injector listens to HTTPS requests on port 443, and the firewall rules of that port must be configured accordingly:

Inbound

FromToPortDescription

Kubernetes Master(s)

Kubernetes Nodes

443

Allow incoming requests from sidecar-injector pod(s)

Outbound

FromToPortDescription

Kubernetes Nodes

Kubernetes Master(s)

443

Allow kubernetes master to communicate with sidecar-injector pod(s)

Installing self-signed certificates on OpenShift

This procedure requires a context with the cluster-admin privilege to create the MutatingWebhookConfiguration.

Follow all of the following instructions in the order listed to ensure that sidecar injector works correctly with your self-signed certificates:

Creating a certificate bundle on OpenShift

These instructions assume that you are working in the project where CloudBees CI is installed, and that the certificate you want to install is named mycertificate.pem.

If you are using a self-signed certificate, add the certificate itself. If you are using a certificate issued by a custom root CA, add the root CA itself.

  1. Copy reference files locally:

    $ oc cp cjoc-0:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ./ca-certificates.crt
    $ oc cp cjoc-0:/etc/pki/ca-trust/extracted/java/cacerts ./cacerts

    On earlier versions, or when overriding the Docker image selection to use the Alpine base:

    $ oc cp cjoc-0:/etc/ssl/certs/ca-certificates.crt ./ca-certificates.crt
    $ oc cp cjoc-0:/etc/ssl/certs/java/cacerts ./cacerts
  2. Add the root CA to the system certificate bundle:

    $ cat mycertificate.pem >> ca-certificates.crt
  3. Add the root CA to the Java cacerts:

    $ keytool -import -noprompt -keystore cacerts -file mycertificate.pem -storepass changeit -alias service-mycertificate;
    Make sure that mycertificate.pem contains only one certificate. keytool does not support importing multiple certificates from a single file.
  4. Use the two files above to create a configmap for each labeled namespace that uses sidecar injector:

    $ oc create configmap --from-file=ca-certificates.crt,cacerts ca-bundles

Setting up the sidecar injector on OpenShift using Helm install

This procedure shows you how to set up sidecar injector using the helm install method. If you are unable to use helm install, go to Setting up sidecar injector on OpenShift using Helm template.

  1. Create a project to deploy the sidecar injector:

    $ oc new-project cloudbees-sidecar-injector
  2. Type the following command:

    helm install cloudbees-sidecar-injector cloudbees/cloudbees-sidecar-injector --namespace cloudbees-sidecar-injector
  3. Verify that the sidecar-inject-webhook pod is running:

    $ oc --namespace cloudbees-sidecar-injector get pods
    NAME                                                  READY     STATUS    RESTARTS   AGE
    sidecar-injector-webhook-deployment-bbb689d69-882dd   1/1       Running   0          5m
    
    $ oc --namespace cloudbees-sidecar-injector get deployment
    NAME                                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    sidecar-injector-webhook-deployment   1         1         1            1           5m

Setting up the sidecar injector on OpenShift using Helm template

Use this procedure to set up sidecar injector if you are unable to use helm install. This procedure uses the helm template method.

  1. Create a project to deploy the sidecar injector:

    $ oc new-project cloudbees-sidecar-injector
  2. Enter the following command:

    helm template cloudbees-sidecar-injector cloudbees/cloudbees-sidecar-injector --namespace cloudbees-sidecar-injector --api-versions "route.openshift.io/v1" > sidecar-injector.yaml
  3. Enter the following command:

    oc apply --namespace cloudbees-sidecar-injector -f sidecar-injector.yaml.
  4. Verify that the sidecar-inject-webhook pod is running:

    $ oc --namespace cloudbees-sidecar-injector get pods
    NAME                                                  READY     STATUS    RESTARTS   AGE
    sidecar-injector-webhook-deployment-bbb689d69-882dd   1/1       Running   0          5m
    
    $ oc --namespace cloudbees-sidecar-injector get deployment
    NAME                                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    sidecar-injector-webhook-deployment   1         1         1            1           5m

Configuring a namespace for sidecar injector on OpenShift

  1. Label the namespace where CloudBees CI is installed with sidecar-injector=enabled:

    $ oc label namespace mynamespace sidecar-injector=enabled
  2. Check the configuration:

    $ oc get namespace -L sidecar-injector
    NAME          STATUS    AGE       SIDECAR-INJECTOR
    default       Active    18h
    mynamespace   Active    18h       enabled
    kube-public   Active    18h
    kube-system   Active    18h

Verifying the namespace for sidecar injector on OpenShift

  1. Switch to the CloudBees CI project:

    $ oc project mynamespace
  2. Deploy an app in the OpenShift cluster. For example, the sleep app:

    # cat <<EOF | oc create -f -
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: sleep
    spec:
    replicas: 1
    template:
      metadata:
    	labels:
    	  app: sleep
      spec:
    	containers:
    	- name: sleep
    	  image: tutum/curl
    	  command: ["/bin/sleep","infinity"]
    EOF
  3. Verify that injection has occurred:

    $ oc get pods -o 'go-template={{range .items}}{{.metadata.name}}{{"\n"}}{{range $key,$value := .metadata.annotations}}* {{$key}}: {{$value}}{{"\n"}}{{end}}{{"\n"}}{{end}}'
          sleep-d5bf9d8c9-bfglq
          * com.cloudbees.sidecar-injector/status: injected

Applying the certificate bundle

After you have completed the setup for sidecar injector, you can use your custom CA across your cluster.

To apply the new certificate bundle:

  1. Use CJOC_RUL/restart to restart operations center and the pod.

  2. Restart any running Managed Masters.

When new build agents are scheduled, the certificate bundle is automatically applied and permits connection to remote endpoints using your certificates.

Advanced configuration for sidecar injector

Disabling Injection on a specific pod

To disable implicit injection for a specific pod, annotate it with com.cloudbees.sidecar-injector/inject: no

Making injection explicit

By default, injection is implicit and applies to all pods created in the labelled namespace(s). However, you can alternately enable injection on only the pod(s) that explicitly require it.

To make injection explicit for a given pod:

  1. Edit the sidecar-injector-webhook-configmap configmap and specify requiresExplicitInjection: true.

  2. To enable injection on a specific pod, annotate the pod with com.cloudbees.sidecar-injector/inject: yes.