Use self-signed certificates in CloudBees CD/RO on Kubernetes

CloudBees CD/RO includes an option called sidecar injector. This option lets you use a self-signed certificate or a custom certificate authority (CA) to access internal HTTPS services, such as an SCM repository or an artifact repository.

Sidecar injector is designed only to trust services that are secured with custom or self-signed certificates. It is not intended to be used to secure a CloudBees CD/RO cluster using HTTPS and should not be set up for that purpose.

Using sidecar injector provides the following benefits:

Satisfies the enterprise security policies for organizations that use custom certificate authorities.
Removes the cost of purchasing commercially signed certificates for internal websites.
Saves time because organizations don’t have to wait for signed certificates.

To use sidecar injector, create a certificate bundle that overwrites the default certificate bundle. Then, set up sidecar injector to inject that certificate bundle into all containers of all scheduled Kubernetes pods in a labeled namespace.

Prerequisites for using self-signed certificates on Kubernetes

Sidecar injector requires a currently supported version of Kubernetes. For more information, refer to Supported platforms for CloudBees CD/RO on Kubernetes.

Sidecar injector version 2.2.0 requires Kubernetes version 1.21 or later. If you run one of the older supported versions of Kubernetes, you should install sidecar injector version 2.1.3. However, to perform a Helm installation requires sidecar injector version 2.2.0 or later.

The prerequisites for using self-signed certificates on Kubernetes are as follows:

Certificate files to install using this feature.
The MutatingAdmissionWebhook admission controller must be enabled.

To check whether it is enabled for your cluster, start by running the following command:
```
kubectl api-versions | grep admissionregistration.k8s.io/v1beta1▼
```
The result should be:
```
admissionregistration.k8s.io/v1▼
```
or
```
admissionregistration.k8s.io/v1
admissionregistration.k8s.io/v1beta1▼
```
This means the APIs are available on your cluster.
In addition, you must add the MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers and list them in the correct order in the enable-admission-plugins flag of kube-apiserver. The way to check this depends on the Kubernetes distribution.

For public cloud offerings, such as Amazon EKS, AKS, or GKE, they are enabled. For other distributions, check the corresponding documentation or ask the provider to determine whether the admission controllers are available.

When your prerequisites are complete, you are ready to begin installing self-signed certificates on Kubernetes.

Installing self-signed certificates on Kubernetes

This procedure requires a context with cluster-admin privilege to create the MutatingWebhookConfiguration.

Sidecar Injector is delivered as a Helm chart. It can be installed either directly using Helm or as a yaml manifest produced by helm template.

You may have previously used a different method for setting up self-signed certificates. If so, please refer to our article detailing that process to undo those changes before continuing here.

Creating a certificate bundle

These instructions assume you are working in the namespace where CloudBees CD/RO is installed, and that the certificate you want to install is named mycertificate.pem.

If you are using a self-signed certificate, add the certificate itself. You can download the certificate using the following command (change www.example.com to the site that is not trusted by the JVM):

openssl s_client -showcerts -connect www.example.com:443 </dev/null 2>/dev/null | openssl x509 -outform PEM > mycertificate.pem▼

If the certificate has been issued from a custom root CA, add the root CA itself.

Create a ca-bundles config map using your certificates by running the following:

kubectl create configmap --from-file=ca-certificates.crt,cacerts ca-bundles -n mynamespace▼

If you already have CloudBees CD/RO running, get pods using:

kubectl get pods -n mynamespace▼

Sample output:

kubectl get pods -n mynamespace
NAME                                                            READY   STATUS    RESTARTS   AGE
cloudbees-flow-nginx-ingress-controller-5879888789-qwzdz        1/1     Running   0          34m
cloudbees-flow-nginx-ingress-default-backend-7f4c876765-sc9cd   1/1     Running   0          33m
flow-bound-agent-6c4678cfbd-fkr84                               1/1     Running   0          33m
flow-devopsinsight-0                                            1/1     Running   1          60m
flow-repository-7dbd4c75b7-llzq9                                1/1     Running   0          34m
flow-server-97674cf77-z84mm                                     1/1     Running   0          35m
flow-web-795bb886d6-gg4pf                                       1/1     Running   0          35m
mariadb-0                                                       1/1     Running   0          60m▼

Run the commands below to generate certificates:
```
mkdir ca-certs-test
cd ca-certs-test▼
```
Use a flow-server pod, for example: flow-server-97674cf77-z84mm.

Run the command below to retrieve the Unix certificates file.

kubectl cp flow-server-97674cf77-z84mm:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ./ca-certificates.crt▼

Run the command below to retrieve the Java CA certificates file.

kubectl cp flow-server-97674cf77-z84mm:/opt/cbflow/jre/lib/security/cacerts ./cacerts▼

Add root CA to system certificate bundle by running the following:
```
cat mycertificate.pem >> ca-certificates.crt▼
```
Add root CA to Java cacerts by running the following:
```
keytool -import -noprompt -keystore cacerts -file mycertificate.pem -storepass changeit -alias service-mycertificate;▼
```
Verify that mycertificate.pem contains only one certificate. keytool does not support importing multiple certificates from a single file.
Create a configmap with the two files above in your namespace (change mynamespace in this command to be the namespace you would like to use sidecar injector with):
```
kubectl create configmap --from-file=ca-certificates.crt,cacerts ca-bundles -n mynamespace▼
```
You have created the certificate bundle. You are now ready to install the CloudBees CD/RO and Kubernetes cluster.

Installing CloudBees CI and CloudBees CD/RO in the same Kubernetes cluster with sidecar enabled

If you already have CloudBees CI and CloudBees CD/RO installed in the same Kubernetes cluster with sidecar enabled, it is not necessary to enable sidecar injector for CloudBees CD/RO.

Create a ca-bundles config map. For more information, refer to Creating a certificate bundle.

Sidecar injector supports the ca-certificates and Java keystore paths by default. To mount additional paths, you must update your CloudBees CI deployment with additional paths in the sidecarinjector: section of the Helm chart values file and upgrade it.

injectionCaCertificates:
  - /etc/ssl/certs/ca-certificates.crt # Alpine/Debian/Ubuntu/Gentoo etc.
  - /etc/pki/tls/certs/ca-bundle.crt # Fedora/RHEL 6
  - /etc/ssl/ca-bundle.pem # OpenSUSE
  - /etc/pki/tls/cacert.pem # OpenELEC
  - /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem # CentOS/RHEL 7
# Files to inject as Java keystore. Some entries can be removed if linux distributions are known in advance
injectionJavaKeystore:
  - /etc/ssl/certs/java/cacerts # Alpine/Debian/Ubuntu/Gentoo etc.
  - /var/lib/ca-certificates/java-cacerts # OpenSUSE
  - /etc/pki/java/cacerts # CentOS/RHEL 6-7
  - /opt/cbflow/jre/lib/security/cacerts▼

Sidecar injector automatically mounts your certificates when it finds the paths and valid namespace after you restart your pods.

Run the following commands to scale up and scale down your flow-server deployment if you are upgrading an existing installation to mount certificates.

kubectl scale deployment flow-server --replicas=0 -n mynamespace
kubectl scale deployment flow-server --replicas=1  -n mynamespace
kubectl get pods -n mynamespace▼

Run the commands below to verify injection.

kubectl get pods -n mynamespace▼

kubectl describe pods pods flow-server-97674cf77-z84mm -n mynamespace▼

Verify the paths below in the Mounts section appear as mounted.

/opt/cbflow/jre/lib/security/cacerts

/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

You have installed CloudBees CI and CloudBees CD/RO in the same Kubernetes cluster with sidecar enabled. You are now ready to set up the sidecar injector on Kubernetes using Helm.

Setting up the sidecar injector on Kubernetes using Helm

This procedure sets up sidecar injector on Kubernetes using Helm.

To set up sidecar injector using Helm:

Run the command below to create a namespace to deploy the sidecar injector.
```
kubectl create namespace cloudbees-sidecar-injector▼
```

Install sidecar-injector using one of the following:

The Helm command:

helm repo update
helm install cloudbees-sidecar-injector cloudbees/cloudbees-sidecar-injector --namespace cloudbees-sidecar-injector▼

The Helm template command (depending on the Kubernetes version, you may also need to pass --api-versions for your targeted Kubernetes API versions, as per https://helm.sh/docs/helm/helm_template/):

helm template cloudbees-sidecar-injector cloudbees/cloudbees-sidecar-injector --namespace cloudbees-sidecar-injector | kubectl apply -n cloudbees-sidecar-injector -f -▼

Run the command below to verify everything is running:

kubectl --namespace cloudbees-sidecar-injector get pods▼

The cloudbees-sidecar-injector pod should be running, and the output should be similar to:

NAME                                                  READY     STATUS    RESTARTS   AGE
cloudbees-sidecar-injector-bbb689d69-882dd   1/1       Running   0          5m▼

Run the command below to verify that the deployment has one pod running and is up-to-date:
```
kubectl --namespace cloudbees-sidecar-injector get deployment▼
```
The output should be similar to:
```
NAME                                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
cloudbees-sidecar-injector   1         1         1            1           5m▼
```
Setup of sidecar injector on Kubernetes using Helm is complete. You are now ready to configure a namespace for sidecar injector on Kubernetes.

Configuring a namespace for sidecar injector on Kubernetes

This process configures a namespace for sidecar injector on Kubernetes.

Label the namespace where CloudBees CD/RO is installed with sidecar-injector=enabled.

In the command below, replace "mynamespace" with the name of your namespace. For example, if your namespace is "cb", use the command kubectl label namespace cb sidecar-injector=enabled. This causes pods that start in that namespace to have the sidecar injected.

If you configure sidecar injector to use its own namespace, make sure that the namespace does not contain the sidecar-injector=enabled label.

kubectl label namespace mynamespace sidecar-injector=enabled▼

Check the following:

kubectl get namespace -L sidecar-injector▼

The output should be similar to:

NAME                       STATUS    AGE       SIDECAR-INJECTOR
cloudbees-sidecar-injector Active    1h
default                    Active    18h
kube-public                Active    18h
kube-system                Active    18h
mynamespace                Active    18h       enabled▼

Verifying the namespace for sidecar injector on Kubernetes

This process allows you to verify the namespace for sidecar injector on Kubernetes.

Deploy an app in the Kubernetes cluster, take sleep app as an example. Run the command below.

kubectl run sleep -n mynamespace --generator=run-pod/v1 --image alpine --serviceaccount default --command /bin/sleep infinity▼

Run the following command to verify injection.

kubectl get pods -n mynamespace -o 'go-template={{range .items}}{{.metadata.name}}{{"\n"}}{{range $key,$value := .metadata.annotations}}* {{$key}}: {{$value}}{{"\n"}}{{end}}{{"\n"}}{{end}}'▼

The output should be similar to:

sleep
* com.cloudbees.sidecar-injector/status: injected▼

Run the command below to delete the sleep pod.
```
kubectl delete pod sleep -n mynamespace▼
```
You have verified the namespace for sidecar injector on Kubernetes.

Advanced configuration for sidecar injector on Kubernetes

Disabling injection on a specific pod

To disable implicit injection for a specific pod, annotate it with com.cloudbees.sidecar-injector/inject: no.

Making injection explicit

By default, injection is implicit and applies to all pods created in the labeled namespace(s). However, you can alternately enable injection on only the pod(s) that explicitly require it.

To make injection explicit for a given pod:

Edit the sidecar-injector-webhook-configmap configmap and specify requiresExplicitInjection: true.
To enable injection on a specific pod, annotate the pod with com.cloudbees.sidecar-injector/inject: yes.

Troubleshooting self-signed certificates on Kubernetes

The sleep pod can’t be created

There is an error on pod creation, such as "certificate signed by unknown authority," or the sidecar-injector logs contain the following: http: TLS handshake error from aaa.bbb.ccc.ddd:nnnnn: remote error: tls: bad certificate.

This can happen if the API server TLS certificate differs from the cluster signing certificate. To fix this issue, you must provide the cluster signing certificate as an input to the installation.

Create a values.yaml file as follows. Replace the content with your own certificate.

caBundleCrt: |-
  -----BEGIN CERTIFICATE-----
  MIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
  cm5ldGVzMB4XDTE5MDcyOTA3MzQyMloXDTI5MDcyNjA3MzQyMlowFTETMBEGA1UE
  AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALgQ
  fjIm8iVEpmX3tlcUzuH2BAQGzK0utC6S0hhqnK6zlQ9iGDAwDpAiwGB9VzcJSfK7
  fr3rx4zT9rWfVeot+ARQV/NPxSUpPlGK8WsRleg5wKyUdnE1xKZDly2l2Vpqlr0J
  GnMXb2A0roi685XZo6iQALLfo+rtWQ2y2JLXzGYYCB1sAUX3hM3qbYmIMReBIyMX
  YGUUdaMuWU1YazKy3eJ84Am7l9ZXlMm7infJlAFsM3BCKed9ZxO2KxTvhWv1qbUk
  Bj3GJrL2bJfQi3B6h0piiBDt6YeI3U8yU4EyxtMKQwQXs9T1zHloc6RmVGYYVAEl
  HquW/XIk4ebWTxYND6UCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
  /wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBABsJvnyo14a8FR6y4JGSX1SXSmtS
  uRiWH6qo5Ou+4zIw0LIhDQTtaN44G86BzhvQFdeQzfvyrKXfuYUTOQz/iYPNFsO6
  FOcDg9EcA19n5tSGp8SniyDEe6EhBWa5A9UR2RPEg/8NZRoeZ/2G9SjUqoa/Erxn
  IPlZvNu+gMEK7etysUQ33s4fp+jD6p0pbWKgSQAiDRVHi3Khlhcn7DfM0ncrSQBs
  vgFPSEczjpl8LR6c0pLSdPUHgdK6pDLTYdtdytRNfJAVjAREYL4uSb8I5NKdkgEz
  BLvnDfdDoCeOZoaeLR68jCltNfdzT5/d1v086i7uRepGhQ5w7ehtPuZ+0U8=
  -----END CERTIFICATE-----▼

Then, run the installation again.

helm delete cloudbees-sidecar-injector
helm install cloudbees-sidecar-injector cloudbees/cloudbees-sidecar-injector --namespace cloudbees-sidecar-injector --values values.yaml▼

The sleep pod is created but stays in `ContainerCreating` state

This process allows you to create a sleep pod that is not in the ContainerCreating state.

To create the sleep pod:

Describe the pod by running the following:

kubectl describe po sleep▼

You may get an error like the following:

Warning  FailedMount  2s (x3 over 3s)  kubelet, docker-desktop  MountVolume.SetUp failed for volume "bundles" : configmap "ca-bundles" not found▼

If so, verify that you created the certificate bundle mentioned above. Also check that you are working in the expected namespace.
You must create a configmap named ca-bundles in each namespace with which you want to use sidecar injector.

You have created a sleep pod that is not in the ContainerCreating state.