Controller does not start due to invalid Ingress object

3 minute read

Issue

  • Starting / Restarting a Managed or Team controller from Operations center fails with:

      Failure executing: DELETE at: https://10.55.240.1/apis/networking.k8s.io/v1beta1/namespaces/cbci/ingresses. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:cbci:cjoc" cannot deletecollection resource "ingresses" in API group "networking.k8s.io" in the namespace "cbci".

    Or

      Failure executing: DELETE at: https://10.55.240.1/apis/extensions/v1beta1/namespaces/cbci/ingresses. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. ingresses.extensions is forbidden: User "system:serviceaccount:cbci:cjoc" cannot deletecollection resource "ingresses" in API group "extensions" in the namespace "cbci".

    Or

      Failure executing: POST at: https://10.55.240.1/apis/networking.k8s.io/v1beta1/namespaces/cbci/ingresses. Message: Ingress.extensions "" is invalid: [metadata.name: Required value: name or generateName is required, spec: Invalid value: []networking.IngressRule(nil): either `backend` or `rules` must be specified]. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=metadata.name, message=Required value: name or generateName is required, reason=FieldValueRequired, additionalProperties={}), StatusCause(field=spec, message=Invalid value: []networking.IngressRule(nil): either `backend` or `rules` must be specified, reason=FieldValueInvalid, additionalProperties={})], group=extensions, kind=Ingress, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Ingress.extensions "" is invalid: [metadata.name: Required value: name or generateName is required, spec: Invalid value: []networking.IngressRule(nil): either `backend` or `rules` must be specified], metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

Environment

Related Issues

  • CPLT2-6739: Upgrade K8s Client API - fixed in Master Provisioning Plugin 2.5.10

  • CPLT2-6825: Replace extensions.Ingress by `networking.v1beta1.Ingress` - fixed in Master Provisioning Plugin 2.5.13

Explanation

When customizing the Ingress object via YAML in a Managed controller configuration, the custom YAML:

  • is used to create an Ingress object with the defaults of the Kubernetes Client used by the Master Provisioning Plugin

  • is then merged with the default Ingress object generated by the Master Provisioning Plugin

When those 2 ingress objects are using a different apiVersion, this results in a YAML with 2 Ingress objects, one being incomplete.

The general recommendation is to not specify an apiVersion when customizing controller default resources to avoid this problem. However, between controller Provisioning 2.1.10 and 2.1.13 excluded there was a bug introduced that cause apiVersion of the custom Ingress object and the controller Ingress object to not be in sync. In which case, the apiVersion should be explicitly set in the custom YAML to workaround the problem.

Chronology

In Master Provisioning Plugin < 2.5.10 (CloudBees CI 2.249.2.1):

  • the Kubernetes client used by the plugin defaults to extensions.v1beta for controller custom Ingress resources

  • the plugin explicitly uses extensions.v1beta for controller default Ingress resources

In Master Provisioning Plugin version 2.5.10 (CloudBees CI 2.249.2.1) to 2.5.12 (CloudBees CI 2.249.2.4), the apiVersion were not in sync:

  • the Kubernetes client used by the plugin defaults to networking.k8s.io/v1beta1 for controller custom Ingress resources

  • the plugin explicitly uses extensions.v1beta for controller default Ingress resources

Since Master Provisioning Plugin version 2.5.13 (CloudBees CI 2.249.3.1):

  • the Kubernetes client used by the plugin defaults to networking.k8s.io/v1beta1 for controller custom Ingress resources

  • the plugin explicitly uses networking.k8s.io/v1beta1 for controller default Ingress resources

Resolution

Following section list the solution depending on the versions running:

Master Provisioning Plugin >= 2.5.13 / CloudBees CI >= 2.249.3.1

In the configuration of the Managed controller, remove the apiVersion line of the custom Ingress object in the YAML field and Save.

In the Managed controller page, hit Acknowledge error (and then Free Snapshot for existing controllers). Then Start the Managed controller.

Master Provisioning Plugin 2.5.10 to 2.5.12 / CloudBees CI 2.249.2.x

In the configuration of the Managed controller, add the line apiVersion: "extensions/v1beta1" to the custom Ingress object in the YAML field and Save.

In the Managed controller page, hit Acknowledge error (and then Free Snapshot for existing controllers). Then Start the Managed controller.

Note: When upgrading CloudBees CI to version 2.249.3.1 or later, apiVersion: "extensions/v1beta1" will need to be removed

Master Provisioning Plugin < 2.5.10 / CloudBees CI < 2.249.2.1

In the configuration of the Managed controller, remove the apiVersion line of the custom Ingress object in the YAML field and Save.

In the Managed controller page, hit Acknowledge error (and then Free Snapshot for existing controllers). Then Start the Managed controller.