High Availability Hazelcast discovery doesn’t work if agents are configured in a separate namespace

Issue

You are using CloudBees CI on modern cloud platforms High Availability (active/active) with agents in a separate namespace via the helm values:

Agents:
  SeparateNamespace:
    Enabled: true
    Create: true
    Name: agent-namespace▼

When your replicas start up, you see the error in the controller logs:

WARNING c.hazelcast.kubernetes.KubernetesClient: Kubernetes API access is forbidden! Starting standalone. To use Hazelcast Kubernetes discovery, configure the required RBAC. For 'default' service account in 'default' namespace execute: `kubectl apply -f https://raw.githubusercontent.com/hazelcast/hazelcast/master/kubernetes-rbac.yaml`▼

Environment

Resolution

This issue is tracked as a bug, and will be fixed in an upcoming product release. This article will be updated once the fix is available.

Workaround

Create a Kubernetes role in the namespace where the controller is running (controller-namespace in the example below):

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: controller-hazelcast-discovery-workaround
  namespace: controller-namespace
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list▼

Create a Kubernetes RoleBinding to bind that role to the service account used by controllers (note that jenkins is the default service account name, you may have chosen a different one using rbac.masterServiceAccountName in the Helm values)

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: controller-hazelcast-discovery-workaround
  namespace: controller-namespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: controller-hazelcast-discovery-workaround
subjects:
- kind: ServiceAccount
  name: jenkins▼

After these resources are created, restart or reprovision all replicas of the managed controllers to apply the changes. The Hazelcast discovery should now work as expected, and you will see the replicas under Mange Jenkins > CloudBees CI High Availability.

This workaround should be removed when upgrading to the release with the fix.

Tested product/plugin versions

CloudBees CI on modern cloud platforms - 2.440.1.4