Deploy Self Signed Certificates in controllers and Agents (Custom Location)

Article ID:360018094412
4 minute readKnowledge base

Issue

  • I would like to add / import self signed certificates to JVM truststore of controllers and / or Agents

  • I would like to add / import self signed certificates to use with tool like git and curl

  • Container images do NOT have ca-certificates installed

Environment

Resolution

Current Solution with Sidecar Injector (Post version 2.138.3.1)

Since version 2.138.3.1, the recommended solution is to use a sidecar-injector as explained Using self-signed certificates in CloudBees Jenkins Enterprise.

Otherwise, there is another solution using ConfigMaps like below.

Migrating from ConfigMap to Sidecar Injector

You may have set this up previously and wish to use the current recommended method, or maybe you got started here before finding the new method. If that is the case, you’ll need to undo some changes before following the steps for setting up the sidecar injector. The steps in this article are to remove the certificates based on the below steps for setting them up.

Pre - 2.138.3.1 Solution with ConfigMaps

For version lower than 2.138.3.1, the following solution can be used:

  • 1) Create a ConfigMap that holds cacerts and ca-certificates.crt

  • 2) Mount the ConfigMaps to the Containers that require them - i.e. CJOC, controllers, agents - and set up the pod / containers to point tools to the location of the certificates bundle.

The advantage of this approach is that whenever a new certificates need to be added, it can be done by updating the ConfigMap and restarting the resource that mounts it.

Pre-requisites

  • Kubernetes version 1.10 which provide support for binaries in ConfigMaps - necessary for cacerts

1) Create the ConfigMaps

First, a ConfigMap needs to be created to hold:

  • a preconfigured JVM keystore cacerts

  • a preconfigured CA bundle ca-certificates.crt

Create a directory:

mkdir ca-bundle

Prepare the cacerts

Create / Copy a JVM truststore. You can copy the truststore from an existing JVM installation, for example from the CJOC container.

For CloudBees Core Modern 2.204.1.3 and later:

kubectl cp $CJE_NAMESPACE/cjoc-0:/etc/pki/java/cacerts $(pwd)/ca-bundle/cacerts

For earlier versions:

kubectl cp $CJE_NAMESPACE/cjoc-0:/etc/ssl/certs/java/cacerts $(pwd)/ca-bundle/cacerts

Then import certificates to the cacerts truststore - in this example cje.example.com.pem:

keytool -import -alias cje.example.com -keystore ca-bundle/cacerts -file cje.example.com.pem

Prepare the ca-certificates.crt

Create / Copy a ca-certificates.crt. You can copy the ca-certificates.crt from an existing container, for example from the CJOC container.

For CloudBees Core Modern 2.204.1.3 and later:

kubectl cp $CJE_NAMESPACE/cjoc-0:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem $(pwd)/ca-bundle/ca-certificates.crt

For earlier versions:

kubectl cp $CJE_NAMESPACE/cjoc-0:/etc/ssl/certs/ca-certificates.crt $(pwd)/ca-bundle/ca-certificates.crt

Then add certificates to the ca-certificates.crt. The format should look like the following:

[...]
-----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END CERTIFICATE-----

Create the ConfigMap

The directory ca-bundle should have the following files:

ls ca-bundle/
ca-certificates.crt	cacerts

Create a ConfigMap from that directory:

kubectl create configmap ca-bundle --from-file=ca-bundle -n $CJE_NAMESPACE
configmap "ca-bundle" created

2) Deploy the Certificates

To deploy the certificates:

  • the ConfigMap needs to be mounted to the containers in a specific location - for example /var/certs/

  • environment variables / system properties need to be set at the pod / container level to tools to those locations

In this article we provide example for git with GIT_SSL_CAINFO and curl with CURL_CA_BUNDLE. Other tools might require other environment variables. For example NPM uses NODE_EXTRA_CA_CERTS.

Deploy Certificates in the CJOC

In the cloudbees-core.yaml edit the cjoc statefulset and add:

  • the volume / volumes mounts

  • the system properties -Djavax.net.ssl.trustStore and -Djavax.net.ssl.trustStorePassword to the JAVA_OPTS environment variable

  • the environment variable for the tools using the

Following is an example:

---
apiVersion: "apps/v1beta1"
kind: "StatefulSet"
metadata:
  name: cjoc
  labels:
    com.cloudbees.cje.type: cjoc
    com.cloudbees.cje.tenant: cjoc
spec:
  serviceName: cjoc
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      name: cjoc
      labels:
        com.cloudbees.cje.type: cjoc
        com.cloudbees.cje.tenant: cjoc
    spec:
      serviceAccountName: cjoc
      terminationGracePeriodSeconds: 10
      containers:
      - name: jenkins
        image: cloudbees/cloudbees-cloud-core-oc:2.138.2.2
        env:
        - name: ENVIRONMENT
          value: KUBERNETES
        - name: JENKINS_OPTS
          value: --prefix=/cjoc
        - name: CURL_CA_BUNDLE
          value: /var/certs/ca-certificates.crt
        - name: GIT_SSL_CAINFO
          value: /var/certs/ca-certificates.crt
        - name: JAVA_OPTS
          # To allocate controllers using a non-default storage class, add the following
          # -Dcom.cloudbees.masterprovisioning.kubernetes.KubernetesMasterProvisioning.storageClassName=some-storage-class
          value: >-
            -XshowSettings:vm
            -Djavax.net.ssl.trustStore=/var/certs/cacerts
            -Djavax.net.ssl.trustStorePassword=changeit
            -Dcb.IMProp.warProfiles=kubernetes.json
            -Dcb.IMProp.warProfiles.cje=kubernetes.json
            -Dcom.cloudbees.opscenter.analytics.reporter.JocAnalyticsReporter.PERIOD=120
            -Dcom.cloudbees.opscenter.analytics.reporter.metrics.AperiodicMetricSubmitter.PERIOD=120
            -Dcom.cloudbees.opscenter.analytics.FeederConfiguration.PERIOD=120
            -Dcom.cloudbees.masterprovisioning.kubernetes.KubernetesMasterProvisioning.fsGroup=1000
            -Dhudson.lifecycle=hudson.lifecycle.ExitLifecycle
            -Dcom.cloudbees.jce.masterprovisioning.DockerImageDefinitionConfiguration.disableAutoConfiguration=true
            -Dcom.cloudbees.jce.masterprovisioning.DockerImageDefinitionConfiguration.masterImageName="CloudBees Core - Managed controller 2.138.2.2"
            -Dcom.cloudbees.jce.masterprovisioning.DockerImageDefinitionConfiguration.masterImage=cloudbees/cloudbees-core-mm:2.138.2.2
        volumeMounts:
        - name: volume-ca-bundle
          mountPath: /var/certs
        - name: jenkins-home
          mountPath: /var/jenkins_home
        - name: jenkins-configure-jenkins-groovy
          mountPath: /var/jenkins_config/configure-jenkins.groovy.d
        livenessProbe:
          httpGet:
            path: /cjoc/login
            port: 8080
          initialDelaySeconds: 300
          timeoutSeconds: 5
      volumes:
      - name: jenkins-configure-jenkins-groovy
        configMap:
          name: cjoc-configure-jenkins-groovy
      - name: volume-ca-bundle
        configMap:
          name: ca-bundle
[...]

Apply the config with:

kubectl apply -f cloudbees.core.yaml -n $CJE_NAMESPACE

CJOC should be redeployed automatically.

Deploy Certificates in the Managed controller

Go to Manage Jenkins  Configure System  Kubernetes controller Provisioning  Advanced

Add the Global Properties:

javax.net.ssl.trustStore=/var/certs/cacerts
javax.net.ssl.trustStorePassword=changeit

Add the Global Environment Variables:

CURL_CA_BUNDLE=/var/certs/ca-certificates.crt
GIT_SSL_CAINFO=/var/certs/ca-certificates.crt

Set the YAML:

apiVersion: "apps/v1"
kind: "StatefulSet"
spec:
  template:
    spec:
      containers:
        - name: "jenkins"
          volumeMounts:
          - name: volume-ca-bundle
            mountPath: /var/certs
      volumes:
      - name: volume-ca-bundle
        configMap:
          name: ca-bundle

Important Note: If running CloudBees Core 2.138.1.2 or 2.138.2.2, there is a known issue that requires some extra configuration for this to work. Please have a look at controller Provisioning fails due to an invalid spec.selector

Note: This configuration is only applied to newly created Managed controllers. For existing Managed controllers, the same configuration needs to be applied in the Managed controller configuration and the controller needs to be re-provisioned.

Deploy Certificates in the Agents

Go to the configuration of the Pod Template.

Add Environment Variables to the Pod template:

For Java:

  • Key: JAVA_OPTS

  • Value: -Djavax.net.ssl.trustStore=/var/certs/cacerts -Djavax.net.ssl.trustStorePassword=changeit

For git:

  • Key: GIT_SSL_CAINFO

  • Value: /var/certs/ca-certificates.crt

For tools curl:

  • Key: CURL_CA_BUNDLE

  • Value: /var/certs/ca-certificates.crt

Add Volumes of type ConfigMap Volume to the Pod template:

  • Config Map name: ca-bundle

  • Mount path: /var/certs/

For example:

deploy-ssl-agents-custom.png
This apply the configuration for the jnlp container. If there have several containers that need the certificates, the same can be applied to other containers.

Template Inheritance

When such configuration needs to be applied to multiple agents, Pod Template inheritance can be used.

Troubleshooting

Issue

I see this error when deploying a controller after installing the certificates:

java.io.IOException: Invalid keystore format

Solution

This is most likely due to the keystore not being saved in a binary format in the ConfigMap. Please make sure you are using Kubernetes v1.10 or later as it is required to save the keystore in binary format.

Resources