Using self-signed certificates in CloudBees Core on OpenShift

Prerequisites

OpenShift 3.10 or later, with the MutatingAdmissionWebhook admission controller enabled.

To check whether the admission controller is enabled for your cluster, run the following command:

$ oc api-versions | grep admissionregistration.k8s.io/v1beta1

The result should be:

admissionregistration.k8s.io/v1beta1

When using OpenShift 3.x, add them to the master-config.yaml file as follows:

admissionConfig:
  pluginConfig:
    ValidatingAdmissionWebhook:
      configuration:
        kind: DefaultAdmissionConfig
        apiVersion: v1
        disable: false
    MutatingAdmissionWebhook:
      configuration:
        kind: DefaultAdmissionConfig
        apiVersion: v1
        disable: false

Network requirements

The sidecar injector listens to HTTPS requests on port 443, and the firewall rules of that port must be configured accordingly:

Inbound

From To Port Description

Kubernetes Master(s)

Kubernetes Nodes

443

Allow incoming requests from sidecar-injector pod(s)

Outbound

From To Port Description

Kubernetes Nodes

Kubernetes Master(s)

443

Allow kubernetes master to communicate with sidecar-injector pod(s)

Installing self-signed certificates on OpenShift

This procedure requires a context with the cluster-admin privilege in order to create the MutatingWebhookConfiguration.

The following instructions assume that the working directory is the sidecar-injector subdirectory in the CloudBees Core archive.

Creating a certificate bundle

These instructions assume that you are working in the project where CloudBees Core is installed, and that the certificate you want to install is named mycertificate.pem.

If you are using a self-signed certificate, add the certificate itself. If you are using a certificate issued by a custom root CA, add the root CA itself.

  1. Copy reference files locally:

    $ oc cp cjoc-0:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ./ca-certificates.crt
    $ oc cp cjoc-0:/etc/pki/ca-trust/extracted/java/cacerts ./cacerts

    On earlier versions, or when overriding the Docker image selection to use the Alpine base:

    $ oc cp cjoc-0:/etc/ssl/certs/ca-certificates.crt ./ca-certificates.crt
    $ oc cp cjoc-0:/etc/ssl/certs/java/cacerts ./cacerts
  2. Add the root CA to the system certificate bundle:

    $ cat mycertificate.pem >> ca-certificates.crt
  3. Add the root CA to the Java cacerts:

    $ keytool -import -noprompt -keystore cacerts -file mycertificate.pem -storepass changeit -alias service-mycertificate;
Make sure that mycertificate.pem contains only one certificate. keytool does not support importing multiple certificates from a single file.
  1. Create a configmap using the two files above:

    $ oc create configmap --from-file=ca-certificates.crt,cacerts ca-bundles

Setting up the injector

  1. Browse to the directory where the CloudBees Core archive has been unpacked, and navigate to the sidecar-injector sub-folder.

  2. Create a project to deploy the sidecar injector:

    $ oc new-project sidecar-injector
  3. Create a signed certificate/key pair and store it in an OpenShift secret that will be consumed by the sidecar deployment:

    $ ./webhook-create-signed-cert.sh \
    --service sidecar-injector-webhook-svc \
    --secret sidecar-injector-webhook-certs \
    --namespace sidecar-injector
  4. Patch the MutatingWebhookConfiguration by configuring the caBundle with the correct value from the OpenShift cluster:

    $ cat sidecar-injector.yaml | \
    webhook-patch-ca-bundle.sh > \
    sidecar-injector-ca-bundle.yaml
  5. Switch to the sidecar-injector project:

    $ oc project sidecar-injector
  6. Deploy resources:

    $ oc create -f sidecar-injector-ca-bundle.yaml
  7. Verify that the sidecar-inject-webhook pod is running:

    $ oc get pods
    NAME                                                  READY     STATUS    RESTARTS   AGE
    sidecar-injector-webhook-deployment-bbb689d69-882dd   1/1       Running   0          5m
    
    $ oc get deployment
    NAME                                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    sidecar-injector-webhook-deployment   1         1         1            1           5m

Configuring namespace

  1. Label the namespace where CloudBees Core is installed with sidecar-injector=enabled:

    $ oc label namespace mynamespace sidecar-injector=enabled
  2. Check the configuration:

    $ oc get namespace -L sidecar-injector
    NAME          STATUS    AGE       SIDECAR-INJECTOR
    default       Active    18h
    mynamespace   Active    18h       enabled
    kube-public   Active    18h
    kube-system   Active    18h

Verifying the namespace

  1. Switch to the CloudBees Core project:

    $ oc project mynamespace
  2. Deploy an app in the OpenShift cluster. For example, the sleep app:

    # cat <<EOF | oc create -f -
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: sleep
    spec:
    replicas: 1
    template:
      metadata:
    	labels:
    	  app: sleep
      spec:
    	containers:
    	- name: sleep
    	  image: tutum/curl
    	  command: ["/bin/sleep","infinity"]
    EOF
  3. Verify that injection has occurred:

    $ oc get pods -o 'go-template={{range .items}}{{.metadata.name}}{{"\n"}}{{range $key,$value := .metadata.annotations}}* {{$key}}: {{$value}}{{"\n"}}{{end}}{{"\n"}}{{end}}'
          sleep-d5bf9d8c9-bfglq
          * com.cloudbees.sidecar-injector/status: injected

Conclusion

You can now use your custom CA across your cluster.

To apply the new certificate bundle, restart Operations Center and any running Managed Masters. When new build agents are scheduled, the certificate bundle is automatically applied and permits connection to remote endpoints using your certificates.

Advanced configuration

Disabling Injection on a specific pod

To disable implicit injection for a specific pod, annotate it with com.cloudbees.sidecar-injector/inject: no

Making injection explicit

By default, injection is implicit and applies to all pods created in the labelled namespace(s). However, you can alternately enable injection on only the pod(s) that explicitly require it.

To make injection explicit for a given pod:

  1. Edit the sidecar-injector-webhook-configmap configmap and specify requiresExplicitInjection: true.

  2. To enable injection on a specific pod, annotate the pod with com.cloudbees.sidecar-injector/inject: yes.