Using self-signed certificates in CloudBees CI on Kubernetes

On this page

Prerequisites
Installing self-signed certificates on Kubernetes
Conclusion
Advanced configuration

CloudBees CI includes an optional component called Sidecar Injector. It allows you to use self-signed certificates or a custom root CA (Certificate Authority). It works by injecting a given set of files (certificate bundles) into all containers of all scheduled pods.

Prerequisites

Kubernetes 1.10 or later, with admission controller MutatingAdmissionWebhook enabled.

In order to check whether it is enabled for your cluster, start running the following command:

kubectl api-versions | grep admissionregistration.k8s.io/v1beta1

The result should be:

admissionregistration.k8s.io/v1beta1

This means the APIs are available on your cluster.

In addition, the MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers should be added and listed in the correct order in the admission-control flag of kube-apiserver. The way to check this depends on the Kubernetes distribution.

For public cloud offerings, such as Amazon EKS, AKS or GKE, they are enabled. For other distributions, check the corresponding documentation or ask the provider to determine whether it is available.

Network requirements

The sidecar injector listens to HTTPS requests on port 443, and the firewall rules of that port must be configured accordingly:

Inbound

From	To	Port	Description
Kubernetes Master(s)	Kubernetes Nodes	443	Allow incoming requests from sidecar-injector pod(s)

From

Port

Description

Kubernetes Master(s)

Kubernetes Nodes

443

Allow incoming requests from sidecar-injector pod(s)

Outbound

From	To	Port	Description
Kubernetes Nodes	Kubernetes Master(s)	443	Allow kubernetes master to communicate with sidecar-injector pod(s)

From

Port

Description

Kubernetes Nodes

Kubernetes Master(s)

443

Allow kubernetes master to communicate with sidecar-injector pod(s)

Installing self-signed certificates on Kubernetes

This procedure requires a context with cluster-admin privilege in order to create the MutatingWebhookConfiguration.

Sidecar Injector is delivered as a helm chart. It can be installed either directly using helm or as yaml manifest produced by helm template.

You may have previously used a different method for setting up self-signed certificates. If so, please refer to our article detailing that process to undo those changes before continuing here.

Create a certificate bundle

In the following instructions, we assume you are working in the namespace where CloudBees CI is installed, and the certificate you want to install is named mycertificate.pem.

For a self-signed certificate, add the certificate itself. If the certificate has been issued from a custom root CA, add the root CA itself.

Copy reference files locally

On CloudBees CI 2.204.1.3 or newer:

$ kubectl cp cjoc-0:etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ./ca-certificates.crt
$ kubectl cp cjoc-0:etc/pki/ca-trust/extracted/java/cacerts ./cacerts

On earlier versions, or when overriding the Docker image selection to use the Alpine base:

$ kubectl cp cjoc-0:etc/ssl/certs/ca-certificates.crt ./ca-certificates.crt
$ kubectl cp cjoc-0:etc/ssl/certs/java/cacerts ./cacerts

Add root CA to system certificate bundle

cat mycertificate.pem >> ca-certificates.crt

Add root CA to java cacerts

keytool -import -noprompt -keystore cacerts -file mycertificate.pem -storepass changeit -alias service-mycertificate;

Make sure that mycertificate.pem contains only one certificate. keytool does not support importing multiple certificates from a single file.

Create a configmap with the two files above

kubectl create configmap --from-file=ca-certificates.crt,cacerts ca-bundles

Set up injector

Create a namespace to deploy the sidecar injector.

kubectl create namespace cloudbees-sidecar-injector

Install sidecar-injector using one of the following:

The helm command:

helm install cloudbees-sidecar-injector cloudbees/cloudbees-sidecar-injector --namespace cloudbees-sidecar-injector

The helm template command:

helm template cloudbees-sidecar-injector cloudbees/cloudbees-sidecar-injector --namespace cloudbees-sidecar-injector | kubectl apply -n cloudbees-sidecar-injector -f -

Verify everything is running.

The cloudbees-sidecar-injector pod should be running

$ kubectl --namespace cloudbees-sidecar-injector get pods
NAME                                                  READY     STATUS    RESTARTS   AGE
cloudbees-sidecar-injector-bbb689d69-882dd   1/1       Running   0          5m

$ kubectl --namespace cloudbees-sidecar-injector  get deployment
NAME                                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
cloudbees-sidecar-injector   1         1         1            1           5m

Configure namespace

Label the namespace where CloudBees CI is installed with sidecar-injector=enabled.
```
$ kubectl label namespace mynamespace sidecar-injector=enabled
```

Check the following:

$ kubectl get namespace -L sidecar-injector
NAME                       STATUS    AGE       SIDECAR-INJECTOR
cloudbees-sidecar-injector Active    1h
default                    Active    18h
kube-public                Active    18h
kube-system                Active    18h
mynamespace                Active    18h       enabled

Verify

Deploy an app in Kubernetes cluster, take sleep app as an example.

kubectl run sleep -n mynamespace --generator=run-pod/v1 --image tutum/curl --serviceaccount default --command /bin/sleep infinity

Verify injection has happened.

# kubectl get pods -n mynamespace -o 'go-template={{range .items}}{{.metadata.name}}{{"\n"}}{{range $key,$value := .metadata.annotations}}* {{$key}}: {{$value}}{{"\n"}}{{end}}{{"\n"}}{{end}}'
sleep
* com.cloudbees.sidecar-injector/status: injected

Delete the sleep pod.
```
kubectl delete pod sleep
```

Diagnose possible issues

The sleep pod can’t be created

There is an error on pod creation, such as "certificate signed by unknown authority", or the sidecar-injector logs contain the following http: TLS handshake error from aaa.bbb.ccc.ddd:nnnnn: remote error: tls: bad certificate.

This can happen if the API server TLS certificate differs from the cluster signing certificate. To fix this, you need to provide the cluster signing certificate as an input to the installation.

Create a values.yaml file as follows. Replace the content with your own certificate.

caBundleCrt: |-
  -----BEGIN CERTIFICATE-----
  MIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
  cm5ldGVzMB4XDTE5MDcyOTA3MzQyMloXDTI5MDcyNjA3MzQyMlowFTETMBEGA1UE
  AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALgQ
  fjIm8iVEpmX3tlcUzuH2BAQGzK0utC6S0hhqnK6zlQ9iGDAwDpAiwGB9VzcJSfK7
  fr3rx4zT9rWfVeot+ARQV/NPxSUpPlGK8WsRleg5wKyUdnE1xKZDly2l2Vpqlr0J
  GnMXb2A0roi685XZo6iQALLfo+rtWQ2y2JLXzGYYCB1sAUX3hM3qbYmIMReBIyMX
  YGUUdaMuWU1YazKy3eJ84Am7l9ZXlMm7infJlAFsM3BCKed9ZxO2KxTvhWv1qbUk
  Bj3GJrL2bJfQi3B6h0piiBDt6YeI3U8yU4EyxtMKQwQXs9T1zHloc6RmVGYYVAEl
  HquW/XIk4ebWTxYND6UCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
  /wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBABsJvnyo14a8FR6y4JGSX1SXSmtS
  uRiWH6qo5Ou+4zIw0LIhDQTtaN44G86BzhvQFdeQzfvyrKXfuYUTOQz/iYPNFsO6
  FOcDg9EcA19n5tSGp8SniyDEe6EhBWa5A9UR2RPEg/8NZRoeZ/2G9SjUqoa/Erxn
  IPlZvNu+gMEK7etysUQ33s4fp+jD6p0pbWKgSQAiDRVHi3Khlhcn7DfM0ncrSQBs
  vgFPSEczjpl8LR6c0pLSdPUHgdK6pDLTYdtdytRNfJAVjAREYL4uSb8I5NKdkgEz
  BLvnDfdDoCeOZoaeLR68jCltNfdzT5/d1v086i7uRepGhQ5w7ehtPuZ+0U8=
  -----END CERTIFICATE-----

Then, run the installation again.

helm del --purge cloudbees-sidecar-injector
helm install cloudbees-sidecar-injector cloudbees/cloudbees-sidecar-injector --namespace cloudbees-sidecar-injector --values values.yaml

The sleep pod is created, but stays in `ContainerCreating` state

Describe the pod:

kubectl describe po sleep

You may get an error like the following:

Warning  FailedMount  2s (x3 over 3s)  kubelet, docker-desktop  MountVolume.SetUp failed for volume "bundles" : configmap "ca-bundles" not found

If so, please verify that you created the certificate bundle mentioned above. Also check that you are working in the expected namespace.

Conclusion

You can now use your custom CA across your cluster.

To apply the new certificate bundle, restart Operations Center and any running Managed Masters. When new build agents are scheduled, the certificate bundle is automatically applied and permits connection to remote endpoints using your certificates.

Advanced configuration

Disabling Injection on a specific pod

To disable implicit injection for a specific pod, annotate it with com.cloudbees.sidecar-injector/inject: no

Making injection explicit

By default, injection is implicit and applies to all pods created in the labelled namespace(s). However, you can alternately enable injection on only the pod(s) that explicitly require it.

To make injection explicit for a given pod:

Edit the sidecar-injector-webhook-configmap configmap and specify requiresExplicitInjection: true.
To enable injection on a specific pod, annotate the pod with com.cloudbees.sidecar-injector/inject: yes.