ServiceAccounts created with Terraform have the <code>automounting API credentials</code> disabled by default

Issue

I have deployed CloudBees Core with Terraform but CJOC cannot create controllers. The controller page shows Operation: [get] for kind: [StatefulSet] with name: [controllerName] in namespace: [namespaceName] failed:

Evidences shows that the cjoc pod does not have a volume mounted for the cjoc serviceAccount. kubectl get pod cjoc-0 shows only the following:

[...]
  volumes:
  - name: jenkins-home
    persistentVolumeClaim:
      claimName: jenkins-home-cjoc-0
  - configMap:
      defaultMode: 420
      name: cjoc-configure-jenkins-groovy
    name: jenkins-configure-jenkins-groovy

The Jenkins logs shows PKIX exception:

2019-07-04 04:41:41.245+0000 [id=56]	WARNING	c.c.m.k.KubernetesMasterProvisioning$DescriptorImpl#lambda$getState$2: Caught an exception while retrieving state for KubernetesMasterResource [namespace=null, fsGroup=null, getName()=mm1, getEndpoint()=https://cloudbees-core.example.com/mm1/, getCpus()=1.0, getMemory()=3072.0, getRatio()=0.7, getImage()=DockerImageDefinition{imageTag='cloudbees/cloudbees-core-mm:2.164.3.2', name='CloudBees Core - Managed controller 2.164.3.2'}]
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
Caused: sun.security.validator.ValidatorException: PKIX path building failed
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
	at sun.security.validator.Validator.validate(Validator.java:262)
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
Caused: javax.net.ssl.SSLHandshakeException

Environment

Related Issues

terraform-providers/terraform-provider-kubernetes #263

Explanation

CloudBees Core CJOC and controllers are attached to service accounts so that they can interact with the Kubernetes API to create resources (pods, services, ingresses, etc…). The way this works is that the ServiceAccount credentials are automatically mounted as a Volume to the pod when it starts. The CJOC pod for example should have a volume like the following automatically mounted:

    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: cjoc-token-xxxxx
      readOnly: true

This is the default behavior in Kubernetes if the Service Account admission controller is activated (and it usually is). This behavior can be controlled by a property automountServiceAccountToken as explained in Configure Service Accounts for Pods

The behavior of the Terraform Kubernetes Provider however is different and opt out the automounting API credentials by default on the ServiceAccount object.

Resolution

When using terraform to deploy CloudBees Core, ensure that the cjoc and jenkins service accounts are properly configured with automount_service_account_token = true. For example, like the following assuming CJOC is deployed in the cje namespace

resource "kubernetes_service_account" "cjoc" {
  metadata {
    name = "cjoc"
    namespace = "cje"
  }
  automount_service_account_token = true
}

Refer to the terraform kubernetes provider documentation for more details.

Workaround

A workaround would be to add a volume explicitly to the CJOC / controller StatefulSet that mounts the service account secret.

CJOC

(Note: In those commands, replace <namespace> by the namespace where CJOC is deployed)

1) Get the name of the secret attached to the cjoc service account

kubectl get sa cjoc -n <namespace> -o jsonpath="{.secrets[0].name}"

2) Create a file automount-statefulset-patch.yaml (replace <secret-name> with the name of the secret we just retrieved):

apiVersion: "apps/v1"
kind: "StatefulSet"
spec:
  template:
    spec:
      containers:
      - name: "jenkins"
        volumeMounts:
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: workaround-automount
          readOnly: true
      volumes:
      - name: workaround-automount
        secret:
          defaultMode: 420
          secretName: <secret-name>

3) Apply the patch with:

kubectl patch -p "$(cat automount-statefulset-patch.yaml)" statefulset cjoc -n <namespace>

Master

(Note: In those commands, replace <namespace> by the namespace where the controller is deployed)

1) Get the name of the secret attached to the jenkins service account

kubectl get sa jenkins -n <namespace> -o jsonpath="{.secrets[0].name}"

2) In the configuration of an existing Managed controller, add the following snippet to the YAML field (replace <secret-name> with the name of the secret we just retrieved):

apiVersion: "apps/v1"
kind: "StatefulSet"
spec:
  template:
    spec:
      containers:
      - name: "jenkins"
        volumeMounts:
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: workaround-automount
          readOnly: true
      volumes:
      - name: workaround-automount
        secret:
          defaultMode: 420
          secretName: <secret-name>

Then restart the controller from CJOC’s UI.

The same configuration can be applied under Configure System Kubernetes controller Provisioning Advanced Default Storage Class Name. This setting would apply to newly created controller only.

ServiceAccounts created with Terraform have the automounting API credentials disabled by default