Managed controller hibernation monitor fails with 403 Forbidden on Kubernetes TokenReview API when rbac.install is false

Last Reviewed:2026-06-24()
2 minute readKnowledge base

Issue

After upgrading to CloudBees CI on modern cloud platforms 2.541.3.36069 with Hibernation.Enabled: true and rbac.install: false, the managed-master-hibernation-monitor pod continuously logs 403 Forbidden authentication errors and managed controller hibernation stops functioning. The pod remains in Running state with no restarts, making the failure difficult to detect.

The following error appears in the managed-master-hibernation-monitor pod logs:

WARN [com.clo.ci.aut.ser.KubernetesTokenAuthenticator] (executor-thread-1)
Error authenticating token:
io.fabric8.kubernetes.client.KubernetesClientException:
Failure executing: POST at: https://<kubernetes-api>:443/apis/authentication.k8s.io/v1/tokenreviews.
Message: tokenreviews.authentication.k8s.io is forbidden:
User "system:serviceaccount:<namespace>:managed-master-hibernation-monitor"
cannot create resource "tokenreviews" in API group "authentication.k8s.io"
at the cluster scope.
Received status: Status(code=403, reason=Forbidden ...)
▼

This prevents the hibernation monitor from authenticating tokens, causing managed controller hibernation to fail silently.

Environment

Explanation

Version 2.541.3.36065 introduced KubernetesTokenAuthenticator as a new authentication mechanism in the managed-master-hibernation-monitor component. This mechanism validates inbound tokens by calling the Kubernetes TokenReview API at the cluster scope.

The CloudBees CI on modern cloud platforms Helm chart includes a dedicated template (managed-master-hibernation-monitor-clusterrolebinding.yaml) that creates a ClusterRoleBinding binding the managed-master-hibernation-monitor service account to the Kubernetes built-in system:auth-delegator ClusterRole. This grants the required create permission on tokenreviews.

This template is only rendered when both of the following Helm values are set:

Hibernation:
  Enabled: true

rbac:
  install: true  # if false, the ClusterRoleBinding is not created
▼

In environments where rbac.install: false — typically where Kubernetes RBAC is managed externally — the ClusterRoleBinding is not created during a Helm upgrade, resulting in the 403 error.

Resolution

Before applying a resolution, confirm the root cause:

  1. Verify the ClusterRoleBinding is missing:

    kubectl get clusterrolebinding | grep hibernation
    ▼

    No output indicates the ClusterRoleBinding was not created.

  2. Confirm rbac.install is set to false in the current Helm values:

    helm get values <release-name> -n <namespace>
    ▼

  3. Confirm the Helm chart did not render the ClusterRoleBinding:

    helm get manifest <release-name> -n <namespace> | grep -A 20 "tokenreview"
    ▼

    No output confirms the ClusterRoleBinding is absent from the installed chart.

Apply one of the following options depending on your RBAC management approach.

Option A: Enable Helm-managed RBAC

If rbac.install can be changed to true, run a Helm upgrade:

helm upgrade <release-name> cloudbees/cloudbees-core \
  --version <chart-version> \
  --namespace <namespace> \
  --reuse-values \
  --set rbac.install=true
▼

The Helm chart automatically creates the required ClusterRoleBinding.

Option B: Apply the ClusterRoleBinding manually

If rbac.install must remain false, request your cluster administrator to apply the following manifest:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: managed-master-hibernation-monitor-<namespace>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: managed-master-hibernation-monitor
    namespace: <namespace>
▼
Replace <namespace> with the namespace where CloudBees CI on modern cloud platforms operations center is deployed. system:auth-delegator is a Kubernetes built-in ClusterRole that already exists in all Kubernetes and OpenShift clusters and does not need to be created.

Apply the manifest:

# Kubernetes
kubectl apply -f <filename>.yaml

# OpenShift
oc apply -f <filename>.yaml
▼

Post-resolution verification

After applying either option, restart the pod and confirm the error no longer appears:

# Kubernetes
kubectl rollout restart deployment/managed-master-hibernation-monitor -n <namespace>
kubectl logs -f deployment/managed-master-hibernation-monitor -n <namespace>

# OpenShift
oc rollout restart deployment/managed-master-hibernation-monitor -n <namespace>
oc logs -f deployment/managed-master-hibernation-monitor -n <namespace>
▼

The 403 Forbidden error should no longer appear in the logs.

References

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.