Build Agents are not getting provisioned when using .Agents.SeparateNamespace

Article ID:360059043232
1 minute readKnowledge base

Issue

  • I am using a separate namespace for build agent pods with the helm attribute .Agents.SeparateNamespace but Build Agents do not get provisioned

  • When trying to provision a build agent pod, the controller shows issues like the following:

      2021-02-01 19:54:15.065+0000 [id=31]	WARNING	o.c.j.p.k.KubernetesCloud#provision: Failed to count the # of live instances on Kubernetes
      io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.1.2.3/api/v1/namespaces/agents-ns/pods?labelSelector=jenkins%3Dslave. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:cloudbees-core:jenkins" cannot list resource "pods" in API group "" in the namespace "cloudbees-core".
          at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:589)
          at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:526)
          at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:492)
          at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:451)
          at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:433)
          at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:151)
          at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:635)
          at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:82)
          at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.getPodsWithLabels(KubernetesCloud.java:635)
          at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:550)

Explanation

This is caused by a change / improvement made to the Kubernetes plugin #835 in version 1.28.0. When the Kubernetes plugin check for existing live instances of agent pods (to calculate if within the limits), it uses the namespace configured in the Kubernetes Cloud. However, CloudBees CI does not explicitly configure the Kubernetes Cloud with the agent namespace when using .Agents.SeparateNamespace.

Resolution

The issue has been resolved in CloudBees CI 2.263.4.1.

Upgrade CloudBees CI to version 2.263.4.1 or later.

Workaround

In Operations Center, in the configuration of the "kubernetes shared cloud" set the "Kubernetes Namespace" to the same value as the one used for the helm attribute .Agents.SeparateNamespace.