Helm deployment fails due to ingress security group conflicts

Article ID:360059893312
2 minute readKnowledge base

Issue

I am using AWS ALB for ingress in an AWS EKS environment. When I try to update the security groups annotation (alb.ingress.kubernetes.io/security-groups) with new security groups, the deployment fails with a message similar to the following:

Warning FailedBuildModel 12m (x68 over 13h) ingress Failed build model due to conflicting securityGroups: [sg-123 sg-abc sg-456] | [old-security-group]

Explanation

The alb.ingress.kubernetes.io/security-groups annotation has exclusive merge behavior, which is defined as follows:

Exclusive: such annotation should only be specified on a single Ingress within IngressGroup or specified with same value across all Ingresses within IngressGroup.

In CloudBees CI, ingress resources (CJOC and managed controller) are part of the same IngressGroup. Therefore, the security group annotation values must match for all ingress resources. When provisioning managed controllers, the default behavior is to propagate all CJOC ingress annotations to the ingress resources created for each managed controller. The issue is when updating the security groups via Helm (using OperationsCenter.Ingress.Annotations value) the new security groups are only applied to the CJOC ingress resource and not the managed controller ingress resources.

Resolution

The solution is to limit exclusive annotations to the CJOC ingress and apply non-exclusive annotations to each Managed controller.

Here is an example set of non-exclusive annotations that should be applied to each managed controller ingress resource:

alb.ingress.kubernetes.io/group.name: <groupName>
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/target-type: ip
kubernetes.io/ingress.class: alb
Any custom (non-exclusive) ingress annotations you’ve applied in your environment should be specified as well.

To prevent propagation of CJOC ingress annotations, specify the non-exclusive annotations under CJOC  Configure controller Provisioning  Ingress Annotations (replacing <groupName> with the group name defined in your environment).

mm provisioning ingress annotations

This will ensure all newly provisioned managed controllers will not include the security group annotation. Existing managed controllers need to be restarted for the annotations to be applied.