Environment
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
-
CloudBees CI (CloudBees Core) on traditional platforms - Client controller
-
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
Explanation
The kubernetes plugin can be configured to connect to a remote Kubernetes cluster and provision kubernetes agents that will connect back to the Jenkins controller. This requires:
Authentication
The Kubernetes Plugin offers different methods to authenticate to a remote Kubernetes cluster:
-
Secret Text Credentials: using a Service Account token (see Kubernetes - Authentication - Service Account Tokens)
-
Secret File Credentials: using a KUBECONFIG file (see Kubernetes - Organizing Cluster Access Using kubeconfig Files)
-
Username/Password Credentials: using a username and password (see Kubernetes - Authentication - Static Password File authentication)
-
Certificate Credentials: using client certificates (see Kubernetes - Manage TLS Certificates in a Cluster)
Authorization
In order to be able to provision kubernetes agents in a specific namespace, the authenticated user must have the required RBAC permissions to provision and manage pods in that namespace.
A Role
and a RoleBinding
configuration is available at https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/service-account.yml as a minimum requirement. Those configuration may be adjusted for extended requirements (such as the creation of PVCs).
Resolution
This article explains how to setup the Kubernetes plugin to authenticate to a remote Kubernetes cluster with a Service Account with Kubernetes RBAC:
-
Namespace:
cloudbees-core
-
Service Account:
jenkins
Kubernetes - Authentication with ServiceAccount
First, resources in kubernetes must be created.
Create a Service Account
Create a Service Account jenkins
in the namespace cloudbees-core
kubectl create serviceaccount jenkins -n cloudbees-core
Retrieve the Service Account token and API Server CA
When a Service Account is created, a secret is automatically generated and attached to it. This secret contains base64 encoded information that can be used to authenticate to the Kubernetes API Server as this ServiceAccount:
-
the Kubernetes API Server CA Certificate
-
the Service Account token
Retrieve the ServiceAccount token with this one liner command (the value will be required to configure Jenkins credentials later on):
kubectl get secret $(kubectl get sa jenkins -n cloudbees-core -o jsonpath={.secrets[0].name}) -n cloudbees-core -o jsonpath={.data.token} | base64 --decode
Retrieve the Kubernetes API Server CA Certificate this one liner command (the value will be required to configure the kubernetes plugin later on):
kubectl get secret $(kubectl get sa jenkins -n cloudbees-core -o jsonpath={.secrets[0].name}) -n cloudbees-core -o jsonpath={.data.'ca\.crt'} | base64 --decode
For more details about those values, have a look at Kubernetes - Authentication - Service Account Tokens |
Kubernetes - Authorization with RBAC
Create a Role
Create a Role jenkins
with the required permissions for the kubernetes plugin in the namespace cloudbees-core
:
cat > jenkins-role.yaml << EOF --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: jenkins rules: - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] EOF kubectl apply -f jenkins-role.yaml -n cloudbees-core
Create a RoleBinding
Create a RoleBinding jenkins
that grants the Role jenkins
to the Service Account cloudbees-core
in the namespace cloudbees-core
cat > jenkins-rolebinding.yaml << EOF --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: jenkins namespace: cloudbees-core roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: jenkins subjects: - kind: ServiceAccount name: jenkins EOF kubectl apply -f jenkins-rolebinding.yaml -n cloudbees-core
CloudBees Core - Configure Jenkins and the Kubernetes Cloud
The kubernetes plugin must be configured to use the ServiceAccount created to authenticate to the remote cluster.
Create the Secret Text credentials
In Jenkins create a Secret Text credentials and paste the decoded value of the ServiceAccount token retrieved above in the Secret field.
(Note: you may also change the Scope to System to make the secret visible to global configuration only)
Configure the Kubernetes Cloud
Under
create a new Kubernetes cloud and configure the authentication:-
Kubernetes URL: the Kubernetes API Server URL (you can get it with
kubectl config view --minify | grep server
) -
Kubernetes server certificate key: the Kubernetes API Server CA retrieved above
-
Kubernetes Namespace:
cloudbees-core
the namespace where the ServiceAccount and the RBAC configuration was configured -
Credentials: the Secret Text credentials created above
-
Finally hit the Test Connection to validate the configuration