Kubernetes Plugin: Authenticate with a ServiceAccount to a remote cluster

Issue

I want to configure Jenkins to provision agents in a remote Kubernetes cluster

Environment

Explanation

The kubernetes plugin can be configured to connect to a remote Kubernetes cluster and provision kubernetes agents that will connect back to the Jenkins controller. This requires:

Authentication

The Kubernetes Plugin offers different methods to authenticate to a remote Kubernetes cluster:

Secret Text Credentials: using a Service Account token (see Kubernetes - Authentication - Service Account Tokens)
Secret File Credentials: using a KUBECONFIG file (see Kubernetes - Organizing Cluster Access Using kubeconfig Files)
Username/Password Credentials: using a username and password (see Kubernetes - Authentication - Static Password File authentication)
Certificate Credentials: using client certificates (see Kubernetes - Manage TLS Certificates in a Cluster)

Authorization

In order to be able to provision kubernetes agents in a specific namespace, the authenticated user must have the required RBAC permissions to provision and manage pods in that namespace.

A Role and a RoleBinding configuration is available at https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/service-account.yml as a minimum requirement. Those configuration may be adjusted for extended requirements (such as the creation of PVCs).

Resolution

This article explains how to setup the Kubernetes plugin to authenticate to a remote Kubernetes cluster with a Service Account with Kubernetes RBAC:

Namespace: cloudbees-core
Service Account: jenkins

Kubernetes - Authentication with ServiceAccount

First, resources in kubernetes must be created.

Create a Service Account

Create a Service Account jenkins in the namespace cloudbees-core

kubectl create serviceaccount jenkins -n cloudbees-core

Retrieve the Service Account token and API Server CA

When a Service Account is created, a secret is automatically generated and attached to it. This secret contains base64 encoded information that can be used to authenticate to the Kubernetes API Server as this ServiceAccount:

the Kubernetes API Server CA Certificate
the Service Account token

Retrieve the ServiceAccount token with this one liner command (the value will be required to configure Jenkins credentials later on):

kubectl get secret $(kubectl get sa jenkins -n cloudbees-core -o jsonpath={.secrets[0].name}) -n cloudbees-core -o jsonpath={.data.token} | base64 --decode

Retrieve the Kubernetes API Server CA Certificate this one liner command (the value will be required to configure the kubernetes plugin later on):

kubectl get secret $(kubectl get sa jenkins -n cloudbees-core -o jsonpath={.secrets[0].name}) -n cloudbees-core -o jsonpath={.data.'ca\.crt'} | base64 --decode

For more details about those values, have a look at Kubernetes - Authentication - Service Account Tokens

Kubernetes - Authorization with RBAC

Create a Role

Create a Role jenkins with the required permissions for the kubernetes plugin in the namespace cloudbees-core:

cat > jenkins-role.yaml << EOF
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
EOF
kubectl apply -f jenkins-role.yaml -n cloudbees-core

Create a RoleBinding

Create a RoleBinding jenkins that grants the Role jenkins to the Service Account cloudbees-core in the namespace cloudbees-core

cat > jenkins-rolebinding.yaml << EOF
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins
  namespace: cloudbees-core
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins
EOF
kubectl apply -f jenkins-rolebinding.yaml -n cloudbees-core

CloudBees Core - Configure Jenkins and the Kubernetes Cloud

The kubernetes plugin must be configured to use the ServiceAccount created to authenticate to the remote cluster.

Create the Secret Text credentials

In Jenkins create a Secret Text credentials and paste the decoded value of the ServiceAccount token retrieved above in the Secret field.

(Note: you may also change the Scope to System to make the secret visible to global configuration only)

Configure the Kubernetes Cloud

Under Manage Jenkins Configure System Cloud create a new Kubernetes cloud and configure the authentication:

Kubernetes URL: the Kubernetes API Server URL (you can get it with kubectl config view --minify | grep server)
Kubernetes server certificate key: the Kubernetes API Server CA retrieved above
Kubernetes Namespace: cloudbees-core the namespace where the ServiceAccount and the RBAC configuration was configured
Credentials: the Secret Text credentials created above
Finally hit the Test Connection to validate the configuration