Ingress Annotations missing during controller provisioning since 2.319.2.5

Issue

I am running Kubernetes 1.21 and CloudBees CI >= 2.319.2.5 and < 2.319.2.9. After stopping / starting controllers, the controllers ingresses miss the annotations that are present in the cjoc ingress
I am running Kubernetes 1.22 and CloudBees CI >= 2.319.2.5. After stopping / starting controllers, the controllers ingresses miss the annotations that are present in the cjoc ingress.
I am running AWS EKS 1.21 using AWS Load Balancer Controller and CloudBees CI >= 2.319.2.5 and < 2.319.2.9. After stopping / starting controllers, controllers are not accessible and respond with HTTP error 404. If you click the gear icon for the controller from the operations center, then click the log, you will see the error:
```
  [Fri Feb 04 02:16:26 UTC 2022][Warning][Ingress][test][FailedBuildModel] Failed build model due to couldn't auto-discover subnets: unable to discover at least one subnet
```
I am running AWS EKS 1.22 using AWS Load Balancer Controller and CloudBees CI >= 2.319.2.5. After stopping / starting controllers, controllers are not accessible and respond with HTTP error 404. If you click the gear icon for the controller from the operations center, then click the log, you will see the error:
```
  [Fri Feb 04 02:16:26 UTC 2022][Warning][Ingress][test][FailedBuildModel] Failed build model due to couldn't auto-discover subnets: unable to discover at least one subnet
```

Explanation

There are 2 known issues that causes this problem:

CloudBees CI version 2.319.2.5 introduced a bug BEE-1208 that prevents the cjoc ingress annotations from being inferred during controller provisioning. This bug is fixed in version 2.319.2.9 of CloudBees CI. version 2.6.12 of the Master Provisioning plugin available since version 2.319.3.3 of CloudBees CI.
CloudBees CI version 2.319.2.5 brings support for Kubernetes 1.22 though a legacy usage of Ingress Networking V1 Beta prevents the cjoc ingress annotations from being inferred during controller provisioning when running over Kubernetes 1.22 or later. This bug is fixed in version 2.6.19 of the Master Provisioning plugin that is available in the CloudBees CI 2.332.2.4.

In an environment where Ingress nginx is used, this usually does not have a noticeable impact. In environments where the AWS Load Balancer controller is used however, this causes the controllers to not be accessible anymore.

The operations center web UI will still be functional, as well as any controllers that have not been stopped or started after the upgrade or installation.

Environment

Related Issue(s)

BEE-1208
BEE-15850

Resolution

Kubernetes 1.22 and later

The solution is to upgrade CloudBees CI to version 2.332.2.4 or later.

Kubernetes 1.21 and earlier

The solution is to upgrade CloudBees CI to version 2.319.3.3 or later.

Workaround

NOTE: This workaround did not work with AWS Load Balancer Controller app version 2.2.0, but it does if you upgrade to app version 2.3.1 (latest at this time) and be sure to set createIngressClassResource: true in the helm values, as documented in the install guide prerequisites.

1 . Run kubectl get ing cjoc -o yaml to get the current annotations on the cjoc ingress.

2 . Copy all of the ingress annotations related to alb.ingress.kubernetes.io (below is an example, yours will be different based upon your environment):

---
apiVersion: "networking.k8s.io/v1"
kind: "Ingress"
metadata:
  annotations:
    alb.ingress.kubernetes.io/actions.root-redirect: '{"Type": "redirect", "RedirectConfig":
      { "Path":"/cjoc/teams-check/", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/group.name: ci-cloudbees-core
    alb.ingress.kubernetes.io/healthcheck-path: "${path}login"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip

3 . From the operations center web UI, click the gear icon for the controller, and then the configure button, and scroll down to the Advanced configuration -> YAML section, and paste the ingress annotations from step 2:

Ingress Annotations missing during controller provisioning yaml

4 . Click Save, then Manage and Stop, then Start the controller. You should see the ingress resoure reconcile in the logs

[Fri Feb 04 17:58:13 UTC 2022][Normal][Ingress][prod][SuccessfullyReconciled] Successfully reconciled

5 . Your controller willl now be available, apply this workaround to your other controllers so they will not encounter this issue when they are stopped and started in the future.

6 . To allow your operations center to provision new controllers until the new product release is out, click on Manage Jenkins -> Configure Controller Provisioning and add the ingress annotations from step 2 to the YAML section of this page and save.

7 . These workarounds should be removed after you have upgraded to the correccted version in the Resolution section of this article.

Tested product/plugin versions

CloudBees CI on modern cloud platforms on EKS with ALB helm chart aws-load-balancer-controller-1.3.3, ALB app version v2.3.1