Troubleshooting deployer service account RBAC permission errors during a Helm upgrade on OpenShift

Symptoms

When upgrading CloudBees CI on modern cloud platforms - operations center or CloudBees CI on modern cloud platforms - managed controller using the cloudbees-core Helm chart on OpenShift, the service account used to run helm upgrade (commonly named deployer or cjoc) may lack permissions that newer chart versions require. Users may experience one or more of the following errors:

The upgrade fails while Helm checks an existing cluster-scoped ClusterRole or ClusterRoleBinding:

Error: UPGRADE FAILED: Unable to continue with update: could not get information about the resource ClusterRoleBinding "cjoc-role-binding-cjoc": clusterrolebindings.rbac.authorization.k8s.io "cjoc-role-binding-cjoc" is forbidden: User "system:serviceaccount:cjoc:deployer" cannot get resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope▼

or, once the get permission has been added:

Error: UPGRADE FAILED: cannot patch "cjoc-master-management-cjoc" with kind ClusterRole: clusterroles.rbac.authorization.k8s.io "cjoc-master-management-cjoc" is forbidden: User "system:serviceaccount:cjoc:deployer" cannot patch resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope▼

The upgrade fails because a pre-existing ClusterRole was not created by Helm:

Error: UPGRADE FAILED: Unable to continue with update: ClusterRole "cjoc-master-management-cjoc" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "cloudbees-core"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "cjoc"▼

A controller provisioned in a separate OpenShift project from the operations center fails to be created, with the operations center log showing:

Failure executing: POST at: https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1/namespaces/team-a-jenkins/rolebindings. Message: rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cjoc:deployer" cannot create resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "team-a-jenkins".▼

The upgrade fails while Helm tries to create or update a namespace-scoped Role:

Error: UPGRADE FAILED: failed to create resource: roles.rbac.authorization.k8s.io "cjoc-master-management" is forbidden: user "system:serviceaccount:cjoc:deployer" (groups=["system:serviceaccounts" "system:serviceaccounts:cjoc" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:["rbac.authorization.k8s.io"], Resources:["rolebindings"], Verbs:["delete"]}▼

A controller provisioned in its own OpenShift project fails to start or to launch agents, with the operations center log showing:

Failure executing: POST at: https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1/namespaces/team-a-jenkins/rolebindings. Message: roles.rbac.authorization.k8s.io "cjoc-agents" not found. Received status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[], group=rbac.authorization.k8s.io, kind=roles, name=cjoc-agents, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=roles.rbac.authorization.k8s.io "cjoc-agents" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=NotFound, status=Failure, additionalProperties={}).▼

Diagnosis/Treatment

Start by listing the permissions currently held by the service account that runs the Helm upgrade (replace cjoc and deployer with the applicable namespace and service account name):

oc auth can-i --list --as=system:serviceaccount:cjoc:deployer▼

The output lists every resource and verb the service account currently holds, at namespace scope. Compare it against the scenarios below to identify which permissions are missing.

Deployer service account is missing cluster-scoped RBAC permissions

If rbac.installCluster is set to true in the Helm values file, Helm checks and manages a cluster-scoped ClusterRole named cjoc-master-management-<namespace> and a ClusterRoleBinding named cjoc-role-binding-<namespace> on every upgrade. This requires the service account running Helm to already hold matching cluster-scope permissions on clusterroles and clusterrolebindings, otherwise the get/patch errors shown above occur.

On OpenShift, rbac.installCluster defaults to false because cluster-scoped RBAC is normally managed by the OpenShift cluster administrators outside of the Helm chart.

Confirm whether rbac.installCluster: true is explicitly set in the Helm values file.
If cluster-scoped resources are not required for the deployment topology (for example, a CloudBees CI on modern cloud platforms - multi-cluster operations center setup is not in use), remove the rbac.installCluster: true override so it falls back to the OpenShift default of false.

If rbac.installCluster: true must be kept, grant the service account running Helm the following cluster-scope permissions, then re-run the upgrade:

rules:
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - clusterroles
  - clusterrolebindings
  verbs:
  - get
  - list
  - watch
  - patch▼

Re-run oc auth can-i --list --as=system:serviceaccount:cjoc:deployer and confirm clusterroles/clusterrolebindings now appear with the added verbs, then re-run the upgrade.

ClusterRole or ClusterRoleBinding "cannot be imported into the current release"

This occurs when a ClusterRole or ClusterRoleBinding with the same name already exists in the cluster but was not created by Helm, so it is missing the app.kubernetes.io/managed-by: Helm label and the meta.helm.sh/release-name / meta.helm.sh/release-namespace annotations Helm uses to track ownership. This typically happens when these resources were created manually (for example, by a cluster administrator) before granting the deployer service account the permissions to manage them.

Do not manually add the missing Helm ownership labels and annotations to the existing resource. Patching Helm-managed resources outside of Helm can cause Helm to lose track of them in future upgrades.
Remove the manually created ClusterRole and ClusterRoleBinding.
Grant the deployer service account the cluster-scope permissions listed in the previous scenario so Helm can create and own these resources itself.
Re-run the upgrade and confirm the ClusterRole and ClusterRoleBinding are recreated with the app.kubernetes.io/managed-by: Helm label:
```
oc get clusterrole cjoc-master-management-cjoc -o jsonpath='{.metadata.labels}'▼
```

Deployer service account is missing namespace-scoped permissions for a controller in a separate OpenShift project

By default, managed controllers are provisioned in the same namespace as the operations center. When a managed controller is provisioned in its own, separate OpenShift project, the deployer service account also needs role and rolebinding permissions scoped to that controller’s namespace, not only to the operations center’s namespace.

Identify every namespace in which controllers are provisioned.
For each of those namespaces, grant the deployer service account the same role/rolebinding permissions it already has in the operations center namespace.
Refer to CloudBees CI on modern cloud platforms - Provision a controller in a different OpenShift project than the operations center for the full list of resources a separate project requires.
Re-run the upgrade or the controller provisioning operation, and confirm the controller reaches a running state without RBAC-related errors in the operations center log.

Helm upgrade fails with "attempting to grant RBAC permissions not currently held"

Kubernetes blocks a service account from creating or updating a Role or ClusterRole that defines a permission the service account does not already hold itself. See Restrictions on role creation or update for details on this Kubernetes RBAC behavior.

The error message lists the exact APIGroups, Resources, and Verbs that are missing, for example rolebindings / delete.

Read the APIGroups, Resources, and Verbs reported in the error message.
Add that exact verb, for the same API group and resource, to the deployer service account’s own Role or ClusterRole at the matching scope (namespace or cluster).
Re-run the upgrade and confirm it completes without a new "not currently held" error for a different verb.

Removing the verb from the chart-managed Role/ClusterRole definition instead of adding it to the deployer service account is not a supported workaround, since a later upgrade can reintroduce it and fail again with the same error.

`cjoc-agents` role not found for a controller provisioned in its own namespace

The cjoc-agents Role and RoleBinding, which grant a controller’s service account permission to provision agents, are only created automatically by the Helm chart in two topologies:

All agents run in the same namespace as the operations center (Agents.SeparateNamespace.Enabled: false), with the controller in that same namespace.
All agents run in a single, separate shared namespace (Agents.SeparateNamespace.Enabled: true).

When controllers are instead provisioned each in their own, separate OpenShift project (agents running alongside each controller, not in the operations center’s namespace or in a shared agents namespace), the cjoc-agents role is not created automatically for that project.

Confirm the controller’s namespace does not already have a cjoc-agents Role and RoleBinding.
Create the cjoc-agents Role and RoleBinding manually in that namespace as part of the controller onboarding process, using the same permissions documented in CloudBees CI on modern cloud platforms - Provision a controller in a different OpenShift project than the operations center.
Repeat this step for every new controller project, since it is not created automatically in this topology.
Restart or re-provision the controller and verify agents can be provisioned.

References

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.