Issue
For CloudBees CI on modern cloud platforms:
-
I would like to add / import self signed certificates to JVM truststore of controllers and / or Agents
-
I would like to add / import self signed certificates to use with tool like
gitandcurl -
Container images have ca-certificates installed
| This article describes the process to follow on Kubernetes based installation, for non Kubernetes based installations please refer to this article |
Resolution
For CloudBees CI version 2.138.3.1 and newer, the recommended solution is to use Sidecar injector as per Using self-signed certificates in CloudBees CI on Kubernetes.
Otherwise, there is another solution using ConfigMaps below.
Migrating from ConfigMap to Sidecar injector
You may have set this up previously and wish to use the current recommended method, or maybe you got started here before finding the new method. If that is the case, you’ll need to undo some changes before following the steps for setting up the Sidecar injector. The steps in this article are to remove the certificates based on the below steps for setting them up.
Solution with ConfigMaps
For versions lower than 2.138.3.1, the following solution can be used:
-
1) Create a ConfigMap that holds
cacertsandca-certificates.crt -
2) Mount the ConfigMaps to the Containers that require them - i.e. operations center, controllers, agents - and set up the pod / containers to point tools to the location of the certificates bundle.
The advantage of this approach is that whenever a new certificates need to be added, it can be done by updating the ConfigMap and restarting the resource that mounts it.
Pre-requisites
-
ca-certificates is installed in the image / containers where certificates need to be imported.
-
Kubernetes version 1.10 which provide support for binaries in ConfigMaps - necessary for
cacerts
1) Create the ConfigMap
First, a ConfigMap needs to be created to hold:
-
a preconfigured JVM keystore
cacerts -
a preconfigured CA bundle
ca-certificates.crt
Create a directory, and define a CI_NAMESPACE environment variable with the namespace that CloudBees CI is deployed in your Kubernetes cluster:
mkdir ca-bundle export CI_NAMESPACE="ci"
Prepare the cacerts
Create / Copy a JVM truststore. You can copy the truststore from an existing JVM installation, for example from the operations center container.
For CloudBees CI 2.204.1.3 and newer:
kubectl cp -n ${CI_NAMESPACE} cjoc-0:etc/pki/ca-trust/extracted/java/cacerts ./ca-bundle/cacerts
For earlier versions:
kubectl cp -n ${CI_NAMESPACE} cjoc-0:etc/ssl/certs/java/cacerts ./ca-bundle/cacerts
Then import certificates to the cacerts truststore - in this example ./example.com.pem:
keytool -import -alias "example.com" -keystore ca-bundle/cacerts -file ./example.com.pem
Prepare the ca-certificates.crt
Create / Copy a ca-certificates.crt. You can copy the ca-certificates.crt from an existing container, for example from the operations center container.
For CloudBees CI 2.204.1.3 and newer:
kubectl cp -n ${CI_NAMESPACE} cjoc-0:etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ./ca-bundle/ca-certificates.crt
For earlier versions:
kubectl cp -n ${CI_NAMESPACE} cjoc-0:etc/ssl/certs/ca-certificates.crt ./ca-bundle/ca-certificates.crt
Then add certificates to the ca-certificates.crt using:
cat mycertificate.pem >> ./ca-bundle/ca-certificates.crt
The format should look like the following:
[...] -----BEGIN CERTIFICATE----- xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -----END CERTIFICATE-----
2) Deploy the Certificates
To deploy the certificate, the ConfigMap needs to be mounted to the containers.
With ca-certificates installed, the cacerts / ca-certificates.crt can be injected in specific locations which depends on the underlying distribution in the container:
-
CentOS 7 / RHEL 7:
/etc/pki/java/cacertsand/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem(such as CloudBees Core since version 2.204.1.3) -
Alpine / Debian / Ubuntu / Gentoo etc.:
/etc/ssl/certs/ca-certificates.crtand/etc/ssl/certs/java/cacerts(such as CloudBees Core since before version 2.204.1.3) -
CentOS 6 / RHEL 6:
/etc/pki/java/cacertsand/etc/ssl/certs/ca-bundle.crt
| If not sure, it could be mounted to all locations. |
Deploy Certificates in the operations center
In the CloudBees CI helm values, add the following to mount the configmap to the operations center.
For CloudBees CI versions 2.204.1.3 and later:
OperationsCenter: ExtraVolumes: - name: volume-ca-bundle configMap: name: ca-bundle ExtraVolumeMounts: - name: volume-ca-bundle mountPath: /etc/pki/java/cacerts subPath: cacerts readOnly: true - name: volume-ca-bundle mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem subPath: ca-certificates.crt
For CloudBees CI versions earlier than 2.204.1.3:
OperationsCenter: ExtraVolumes: - name: volume-ca-bundle configMap: name: ca-bundle ExtraVolumeMounts: - name: volume-ca-bundle mountPath: /etc/ssl/certs/java/cacerts subPath: cacerts - name: volume-ca-bundle mountPath: /etc/ssl/certs/ca-certificates.crt subPath: ca-certificates.crt
After applying the helm values, the operations center needs to be redeployed for the change to take effect.
Deploy Certificates in the managed controller
Go to and set the YAML like the following (example for Debian):
apiVersion: "apps/v1" kind: "StatefulSet" spec: template: spec: containers: - name: "jenkins" volumeMounts: - name: volume-ca-bundle mountPath: /etc/pki/java/cacerts # UBI/RHEL based subPath: cacerts - name: volume-ca-bundle mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem # UBI/RHEL based subPath: ca-certificates.crt # Following for version of CloudBees Core on Modern Platform earlier than 2.204.1.3 - name: volume-ca-bundle mountPath: /etc/ssl/certs/java/cacerts # Alpine / Debian / Ubuntu / Gentoo etc. subPath: cacerts - name: volume-ca-bundle mountPath: /etc/ssl/certs/ca-certificates.crt # Alpine / Debian / Ubuntu / Gentoo etc. subPath: ca-certificates.crt volumes: - name: volume-ca-bundle configMap: name: ca-bundle
| If running CloudBees CI 2.138.1.2 or 2.138.2.2, there is a known issue that requires some extra configuration for this to work. Please have a look at controller Provisioning fails due to an invalid spec.selector* |
| This configuration only applied to newly created managed controller. For existing managed controller, the same configuration needs to be applied in the managed controller configuration and the controller needs to be re-provisioned.* |
Deploy Certificates in the Agents
Go to the configuration of the Pod Template set the Raw yaml for the Pod like the following (example for Debian):
apiVersion: v1 kind: Pod spec: containers: - name: "jnlp" volumeMounts: - name: volume-ca-bundle mountPath: /etc/pki/java/cacerts # UBI/RHEL based subPath: cacerts - name: volume-ca-bundle mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem # UBI/RHEL based subPath: ca-certificates.crt # Following for version of CloudBees Core on Modern Platform earlier than 2.204.1.3 - name: volume-ca-bundle mountPath: /etc/ssl/certs/java/cacerts # Alpine / Debian / Ubuntu / Gentoo etc. subPath: cacerts - name: volume-ca-bundle mountPath: /etc/ssl/certs/ca-certificates.crt # Alpine / Debian / Ubuntu / Gentoo etc. subPath: ca-certificates.crt volumes: - name: volume-ca-bundle configMap: name: ca-bundle
This apply the configuration for the jnlp container. If there have several containers that need the certificates, the same can be applied to other containers, for example:
|
apiVersion: v1 kind: Pod spec: containers: - name: "jnlp" volumeMounts: - name: volume-ca-bundle mountPath: /etc/pki/java/cacerts # UBI/RHEL based subPath: cacerts - name: volume-ca-bundle mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem # UBI/RHEL based subPath: ca-certificates.crt # Following for version of CloudBees Core on Modern Platform earlier than 2.204.1.3 - name: volume-ca-bundle mountPath: /etc/ssl/certs/java/cacerts # Alpine / Debian / Ubuntu / Gentoo etc. subPath: cacerts - name: volume-ca-bundle mountPath: /etc/ssl/certs/ca-certificates.crt # Alpine / Debian / Ubuntu / Gentoo etc. subPath: ca-certificates.crt - name: "dind" - name: volume-ca-bundle mountPath: /etc/pki/java/cacerts # UBI/RHEL based subPath: cacerts - name: volume-ca-bundle mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem # UBI/RHEL based subPath: ca-certificates.crt # Following for version of CloudBees Core on Modern Platform earlier than 2.204.1.3 - name: volume-ca-bundle mountPath: /etc/ssl/certs/java/cacerts # Alpine / Debian / Ubuntu / Gentoo etc. subPath: cacerts - name: volume-ca-bundle mountPath: /etc/ssl/certs/ca-certificates.crt # Alpine / Debian / Ubuntu / Gentoo etc. subPath: ca-certificates.crt volumes: - name: volume-ca-bundle configMap: name: ca-bundle
Template Inheritance
When such configuration needs to be applied to multiple agents, Pod Template inheritance can be used.
Troubleshooting
Issue
I see this error when deploying a controller after installing the certificates:
java.io.IOException: Invalid keystore format
Solution
This is most likely due to the keystore not being saved in a binary format in the ConfigMap. Please make sure you are using Kubernetes v1.10 or later as it is required to save the keystore in binary format.