Verify Kubernetes Gateway API prerequisites

13 minute read
CloudBees Beta

This feature is available as a beta release and is subject to change without notice. CloudBees recommends stringent testing in a development environment and a complete review of the documentation and architecture before using it in production.

Modern Cloud Platforms

This content applies only to CloudBees CI on modern cloud platforms.

CloudBees CI on modern cloud platforms uses Gateway API resources, specifically HTTPRoute, to manage ingress routing. After completing these checks, configure Gateway resources for your CloudBees CI on modern cloud platforms deployment.

This guide assumes you already have a Gateway API controller (such as Istio, Envoy Gateway, or NGINX Gateway Fabric) installed in your cluster. For a list of available conformant Gateway API implementations, refer to the Gateway API implementations documentation.

Gateway API manages HTTP/HTTPS traffic only and does not route remoting over TCP traffic (for example, inbound agent connections or controller-to-operations center communication without WebSockets).

Required TLS certificates and DNS records

CloudBees CI on modern cloud platforms does not manage or configure TLS. You are responsible for TLS termination and certificate management, and must configure them before installing CloudBees CI on modern cloud platforms.

CloudBees CI on modern cloud platforms with Gateway API requires the following prerequisites that you must configure before proceeding:

  • Valid TLS certificates: Provide a valid TLS certificate covering your CloudBees CI on modern cloud platforms domain. For subdomain-based routing, this must be a wildcard certificate (for example, *.cloudbees.example.com).

    Where you store the TLS certificate depends on where TLS termination occurs:

  • DNS resolution: Configure DNS records to resolve to the Gateway’s external IP address or load balancer hostname before installing CloudBees CI on modern cloud platforms. For subdomain-based routing, configure a wildcard DNS record (for example, *.cloudbees.example.com) pointing to the Gateway’s external address.

    With wildcard DNS records (the recommended approach for subdomain-based routing), per-host DNS automation tools like external-dns are not required. A single wildcard record covers all CloudBees CI on modern cloud platforms subdomains.

Verify your Gateway API installation

Complete the following steps to verify that your Gateway API installation meets the prerequisites for CloudBees CI on modern cloud platforms:

Verify Gateway API CustomResourceDefinitions are installed

Ensure the Gateway API CustomResourceDefinitions (CRDs) are installed in your cluster.

To verify that the Gateway API CRDs are installed:

  1. Confirm that Gateway API CRDs are installed in your cluster:

    kubectl get crds gateways.gateway.networking.k8s.io
    • If the CRD exists, the expected output is the CRD name and creation date.

    • If the command returns a not found error, the Gateway API is not installed.

  2. Install a Gateway API controller (which includes the CRDs) before proceeding.

    If you plan to use High Availability (HA) or Horizontal Scaling (HS) controllers, install the Gateway API experimental channel CRDs from the start. The experimental channel includes all standard channel resources, plus additional resources required for session persistence. Installing the experimental channel now avoids replacing standard CRDs later. For details, refer to Verify experimental channel CRDs (HA/HS controllers only).

Verify CRD API version compatibility

Verify that your Gateway API CRDs include the v1 API version:

kubectl get crd gatewayclasses.gateway.networking.k8s.io -o jsonpath='{.status.storedVersions}'(1)
1 Checks which API versions are stored for the GatewayClass CRD.

The expected output must include v1 in the list of stored versions. If the output shows only v1beta1, the CRDs are incompatible with CloudBees CI on modern cloud platforms.

Gateway API v1beta1 is incompatible with CloudBees CI on modern cloud platforms

Your Gateway API CRDs must include the v1 API version. If only v1beta1 is present, upgrade your Gateway API CRDs before proceeding.

Verify experimental channel CRDs (HA/HS controllers only)

Gateway API provides two release channels: standard (stable resources) and experimental (features under development). Session persistence via GEP-1619 (sticky sessions) requires experimental channel CRDs. The experimental channel includes all standard channel resources plus additional resources like TCPRoute and policy extensions.

HA/HS controllers require session persistence features provided by the Gateway API experimental channel.

Verify that the experimental channel CRDs are installed:

kubectl get crd tcproutes.gateway.networking.k8s.io

If the CRD exists, the experimental channel is installed. If the command returns a not found error, only the standard channel is installed. Install the experimental channel CRDs before proceeding.

Session persistence implementation support

CloudBees CI on modern cloud platforms uses the sessionPersistence field on HTTPRoute rules (GEP-1619) to provide sticky sessions for HA controllers.

sessionPersistence is an extended support feature in Gateway API. Not all gateway implementations support it, and those that do not will silently ignore it; the implementation generates no errors but does not enforce sticky sessions.

Gateway implementations are still standardizing support. The following table reflects testing and upstream community reports. It is not exhaustive.

Implementation sessionPersistence support Notes

Envoy Gateway

Supported in version 1.2.0 and later. Tested with version 1.7.1.

NGINX Gateway Fabric

Supported in version 2.4.0 and later.

kgateway

Supported in version 2.1.0 and later.

Istio

Silently ignores the field. For more information, refer to Configure sticky sessions with Istio.

GKE Gateway Controller

Silently ignores the field. Configure sticky sessions using vendor-specific resources.

AKS Application Gateway for Containers

Silently ignores the field. Configure sticky sessions using vendor-specific resources.

Configure sticky sessions with Istio

Istio does not currently support the sessionPersistence field and silently ignores it, but CloudBees CI still adds it to HTTPRoutes. Check sessionPersistence behavior after each Istio upgrade and remove the workaround described below once Istio supports it.

To provide sticky sessions with Istio, add a DestinationRule through the YAML configuration section of the controller provisioning configuration. Resources defined in the YAML section are created on controller deploy and deleted on deprovision.

Add the following YAML to the controller provisioning configuration:

apiVersion: networking.istio.io/v1beta1 kind: DestinationRule metadata: name: ${name}-sticky labels: app.kubernetes.io/managed-by: cloudbees-core sticky-session-service: ${name} spec: host: ${name} trafficPolicy: loadBalancer: consistentHash: httpCookie: name: CBCI_STICKY ttl: 0s path: ${path}

The ${name} and ${path} macros are resolved automatically at deploy time:

  • ${name} resolves to the controller service name.

  • ${path} resolves to the controller context path.

    If using Configuration as Code for the operations center to manage the controller item and have enabled variable resolution for items the ${} section must be escaped to prevent CasC from attempting to look up the value as a secret. This can be achieved by prefixing a ^ to the macro (for example, use ^${name} and ^${path}).

This approach requires Role-Based Access Control (RBAC) permissions to manage destinationrules in the networking.istio.io API group. The operations center ServiceAccount must have these permissions. Apply the following Role and RoleBinding to grant them:

apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: istio-destinationrule-manager namespace: <controller-namespace> (1) rules: - apiGroups: - networking.istio.io resources: - destinationrules verbs: - get - list - watch - create - update - patch - delete --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: istio-destinationrule-manager-binding namespace: <controller-namespace> (1) roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: istio-destinationrule-manager subjects: - kind: ServiceAccount name: <service-account-name> (2) namespace: <oc-namespace> (3)
1 The namespace where the controller is deployed.
2 The operations center ServiceAccount name.
3 The namespace where the operations center is deployed.
GatewayClass versus Gateway

GatewayClass is a cluster-scoped resource template created by your gateway controller (for example, Istio creates an istio GatewayClass). Gateway is a namespace-scoped resource you create to configure specific listeners and routing rules. Always verify that a GatewayClass exists and is accepted before creating Gateway resources.

Verify GatewayClass is accepted

A GatewayClass must be present and in an accepted state before you can create Gateway resources.

To verify the GatewayClass is accepted:

  1. List the available GatewayClasses in your cluster:

    kubectl get gatewayclass

    At least one GatewayClass must appear in the output. The CONTROLLER column identifies which gateway controller manages each GatewayClass.

    Not all controllers create a GatewayClass automatically. For example, AWS Load Balancer Controller requires you to create a GatewayClass manually. If no GatewayClass exists, refer to your controller’s documentation for instructions on creating one.

  2. Verify that the GatewayClass is in an accepted state:

    kubectl get gatewayclass istio -o jsonpath='{.status.conditions[?(@.type=="Accepted")].status}'(1)(2)
    1 Filters the status conditions to check only the Accepted condition type.
    2 If you are using a different gateway controller, replace istio with the name of your GatewayClass.

    The expected output is True. If the output is False or empty, the controller has not accepted the GatewayClass; you must resolve the controller configuration issue before proceeding.

Verify Gateway resource

Your cluster needs a Gateway resource with listeners in an Accepted and Programmed state. The Gateway defines how traffic enters your cluster and which namespaces can attach HTTPRoutes to it.

This example Gateway configuration enables cross-namespace routing with Transport Layer Security (TLS) termination:

apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: cloudbees-gateway namespace: gateway-infra spec: gatewayClassName: istio(1) listeners: - name: https protocol: HTTPS(2) port: 443 hostname: "*.cloudbees.example.com"(3) tls: mode: Terminate(4) certificateRefs: - kind: Secret group: "" name: cloudbees-tls-cert(5) allowedRoutes: namespaces: from: Selector(6) selector: matchLabels: cloudbees.com/gateway-routes: enabled(7)
1 References the GatewayClass validated in Verify GatewayClass is accepted. If you are using a different gateway controller, replace istio with your GatewayClass name.
2 HTTPS protocol required for TLS termination.
3 Wildcard hostname matches all CloudBees CI subdomains.
4 Terminate mode: the Gateway decrypts TLS traffic, backends receive plaintext.
5 References a TLS Secret in the same namespace as the Gateway.
6 CRITICAL: This field defaults to Same, which restricts HTTPRoute attachment to the Gateway’s own namespace only. CloudBees CI on modern cloud platforms deploys workloads across multiple namespaces and requires that this value be set to Selector to enable cross-namespace routing. Without this setting, HTTPRoutes created by CloudBees CI on modern cloud platforms in application namespaces fail to attach to the Gateway.
7 Only namespaces with this label can attach HTTPRoutes to this Gateway. Apply this label to all namespaces that contain operations center or managed controller services. For details, refer to Verify namespace labels.

To verify your Gateway resource:

  1. Verify the Gateway exists and is functional:

    kubectl get gateway <gateway-name> -n <gateway-namespace>(1)
    1 Replace <gateway-name> and <gateway-namespace> with your Gateway name and namespace.
  2. Check that the Gateway is in an Accepted state:

    kubectl get gateway <gateway-name> -n <gateway-namespace> -o jsonpath='{.status.conditions[?(@.type=="Accepted")].status}'

    The expected output is True.

  3. Check that the Gateway is in a Programmed state:

    kubectl get gateway <gateway-name> -n <gateway-namespace> -o jsonpath='{.status.conditions[?(@.type=="Programmed")].status}'

    The expected output is True.

  4. Verify the allowedRoutes configuration:

    kubectl get gateway <gateway-name> -n <gateway-namespace> -o jsonpath='{.spec.listeners[0].allowedRoutes.namespaces.from}'

    The expected output is Selector (and is not Same).

    • If the Accepted condition is not True, you must resolve controller configuration issues before proceeding.

    • If the Programmed condition is not True, issue the following command to check the controller logs and Gateway events:

      kubectl describe gateway <gateway-name> -n <gateway-namespace>

Verify DNS resolution

DNS records must resolve to the Gateway’s external address before CloudBees CI on modern cloud platforms can serve traffic.

To verify DNS resolution:

  1. Retrieve the Gateway’s external address:

    kubectl get gateway <gateway-name> -n <gateway-namespace> -o jsonpath='{.status.addresses[0].value}'

    The expected output is the external IP address or hostname your controller assigned to the Gateway.

  2. Verify that your wildcard DNS record resolves to this address:

    dig +short test.cloudbees.example.com(1)
    1 Replace test.cloudbees.example.com with any subdomain matching your wildcard DNS record. The output must match the Gateway’s external address.

    If DNS does not resolve, create the appropriate DNS record before proceeding. For subdomain-based routing, create a wildcard A record or CNAME record (for example, *.cloudbees.example.com) pointing to the Gateway’s external address.

Verify TLS certificate (Gateway-level TLS termination only)

The Gateway listener references a TLS certificate Secret that must exist in the Gateway namespace. The TLS certificate Secret must be of type kubernetes.io/tls and reside in the same namespace as the Gateway.

This step applies only if TLS termination occurs at the Gateway. If your load balancer handles TLS termination (for example, AWS ALB with ACM certificates), the Gateway does not reference a TLS Secret and you can skip this step.

To verify the TLS certificate configuration:

  1. Retrieve the Secret name from the Gateway:

    kubectl get gateway <gateway-name> -n <gateway-namespace> -o jsonpath='{.spec.listeners[0].tls.certificateRefs[0].name}'(1)
    1 Displays the Secret name referenced by the Gateway listener.
  2. Verify the Secret exists and check its type:

    kubectl get secret <secret-name> -n <gateway-namespace> -o jsonpath='{.type}'(1)

    The expected output is kubernetes.io/tls.

  3. Verify the Secret contains the required keys:

    kubectl get secret <secret-name> -n <gateway-namespace> -o jsonpath='{.data}' | grep -o 'tls\.\(crt\|key\)'(1)
    1 Must show both tls.crt and tls.key.
  4. Verify the TLS mode is set to Terminate:

    kubectl get gateway <gateway-name> -n <gateway-namespace> -o jsonpath='{.spec.listeners[0].tls.mode}'(1)

    The expected output is Terminate.

    If you use cert-manager or another certificate manager, verify the Secret is created in the Gateway namespace.

    • If the Secret type is Opaque instead of kubernetes.io/tls, recreate the Secret with kubectl create secret tls.

    • If the Secret is missing, check your certificate provisioning process.

Verify namespace labels

The Gateway’s allowedRoutes selector acts as a filter — only namespaces with matching labels can create HTTPRoutes through this Gateway. If your organization separates Gateway administration from application namespace management, coordinate with your Gateway administrator to determine the required label. For detailed namespace topology patterns and responsibility boundaries, refer to Deploy a Kubernetes Gateway API namespace topology.

To verify namespace labels:

  1. Retrieve the Gateway’s namespace selector:

    kubectl get gateway <gateway-name> -n <gateway-namespace> -o jsonpath='{.spec.listeners[0].allowedRoutes.namespaces.selector.matchLabels}'(1)
    1 This shows the label key-value pairs required on application namespaces.
  2. Verify your application namespace has the matching label:

    kubectl get namespace <namespace-name> --show-labels(1)
    1 Look for the label key from the previous command.
  3. Apply the label if it is missing:

    kubectl label namespace <namespace-name> cloudbees.com/gateway-routes=enabled(1)
    1 This label must match the matchLabels selector on your Gateway’s allowedRoutes configuration. Apply it to every namespace containing operations center or managed controller services that need HTTPRoutes. Do not label agent namespaces.

    The CloudBees CI on modern cloud platforms Helm chart cannot manage labels on the release namespace because the namespace must exist before helm install runs. You must apply this label using kubectl as a separate step, both for initial installation and upgrades. For namespaces created by the Helm chart (for example, agent namespaces via Agents.SeparateNamespace.Create), the chart manages labels automatically.

  4. Verify the label was applied:

    kubectl get namespace <namespace-name> --show-labels

    If HTTPRoutes fail to attach despite from: Selector being set, verify the namespace label key and value match the Gateway selector exactly. Typos in label keys are a common cause of attachment failures.

Verify RBAC permissions

CloudBees CI on modern cloud platforms needs RBAC permissions to create and manage HTTPRoute resources in the application namespace. HTTPRoute is a custom resource in the gateway.networking.k8s.io API group, so explicit RBAC grants are required.

  • The CloudBees CI on modern cloud platforms Helm chart creates the required RBAC resources (Roles and RoleBindings) for HTTPRoute management automatically. This step verifies that the RBAC permissions are in place and helps diagnose issues if HTTPRoute creation fails. You should not need to create RBAC resources manually unless you have customized your Helm deployment or use a restricted RBAC policy.

  • CloudBees CI on modern cloud platforms operations center and each managed controller require HTTPRoute management permissions in their respective namespaces. Replace <service-account-name> and <namespace> with the values for each component.

To verify RBAC permissions:

  1. Verify that the CloudBees CI ServiceAccount has permission to create HTTPRoute resources:

    kubectl auth can-i create httproutes.gateway.networking.k8s.io \ --as=system:serviceaccount:<namespace>:<service-account-name> \ -n <namespace>(1)
    1 Tests permissions as the CloudBees CI ServiceAccount, not the current user.

    The expected output is yes. If the output is no, apply the Role and RoleBinding below.

  2. Verify all required HTTPRoute management verbs:

    for verb in get list watch create update patch delete; do echo -n "$verb: " kubectl auth can-i $verb httproutes.gateway.networking.k8s.io \ --as=system:serviceaccount:<namespace>:<service-account-name> \ -n <namespace> done

    The expected output is yes for all verbs.

    Running kubectl auth can-i without --as tests your current user’s permissions, not the ServiceAccount permissions that CloudBees CI on modern cloud platforms uses at runtime. Always use the --as=system:serviceaccount:<namespace>:<service-account-name> flag.

    Testing with RBAC groups

    If your operations center uses a Kubernetes RBAC group for permissions instead of a ServiceAccount, use the --as-group flag:

    kubectl auth can-i create httproutes.gateway.networking.k8s.io \ --as test --as-group=<RBAC-group> \(1) -n <namespace>
    1 Replace <RBAC-group> with the Kubernetes RBAC group assigned to your operations center.
  3. If RBAC permissions are missing and the Helm chart did not create them (for example, due to custom Helm values or restricted cluster policies), apply the following Role and RoleBinding:

    apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: httproute-manager namespace: <namespace>(1) rules: - apiGroups: - gateway.networking.k8s.io(2) resources: - httproutes(3) verbs: - get - list - watch - create - update - patch - delete(4) --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: httproute-manager-binding namespace: <namespace>(5) roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: httproute-manager subjects: - kind: ServiceAccount name: <service-account-name>(6) namespace: <namespace>
    1 The namespace where CloudBees CI on modern cloud platforms is deployed.
    2 HTTPRoute resources are in the gateway.networking.k8s.io API group.
    3 Resource name must be lowercase plural: httproutes.
    4 Full lifecycle management verbs.
    5 RoleBinding must be in the same namespace as the ServiceAccount.
    6 ServiceAccount name used by CloudBees CI on modern cloud platforms. Check your Helm values or existing deployments.

    If deploying CloudBees CI on modern cloud platforms across multiple namespaces, consider using a ClusterRole with per-namespace RoleBindings instead of duplicating Roles in each namespace.

  4. After applying the YAML, verify the permissions are granted:

    kubectl auth can-i create httproutes.gateway.networking.k8s.io \ --as=system:serviceaccount:<namespace>:<service-account-name> \ -n <namespace>

    The expected output is yes.

Previously, with Ingress NGINX, you configured HTTP-to-HTTPS redirects and root path redirects through annotations on the Ingress resource (ssl-redirect, app-root). With Gateway API, CloudBees CI on modern cloud platforms no longer manages these behaviors. Platform teams should configure these Gateway-level concerns directly on the Gateway or as standalone HTTPRoute resources.

HTTPS redirect (previously ssl-redirect)

With Ingress NGINX, the ssl-redirect annotation on the Ingress resource handled HTTP-to-HTTPS redirects automatically. With Gateway API, configure this redirect on the Gateway itself, typically by adding an HTTP listener on port 80 with an HTTPRoute that performs a redirect to HTTPS.

CloudBees strongly recommends configuring this redirect to prevent users from accessing CloudBees CI on modern cloud platforms over unencrypted HTTP.

Example HTTP-to-HTTPS redirect HTTPRoute
apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: https-redirect namespace: gateway-infra spec: parentRefs: - name: cloudbees-gateway sectionName: http(1) rules: - filters: - type: RequestRedirect requestRedirect: scheme: https statusCode: 301
1 References an HTTP listener on the Gateway. The Gateway must have a listener named http on port 80 for this route to attach.

Root path redirect (previously app-root)

Previously, with Ingress NGINX, the app-root annotation redirected requests from / to the operations center context path (for example, /cjoc/). With Gateway API, CloudBees CI on modern cloud platforms does not create this redirect.

Platform teams can optionally configure this as an HTTPRoute if users access the root path directly.

Quick reference and troubleshooting

Use this checklist to verify all Gateway API prerequisites before installing CloudBees CI on modern cloud platforms or to troubleshoot issues after installation. Select the prerequisite name to view detailed verification steps and remediation guidance.

Prerequisite Quick check Expected result
kubectl get crds gateways.gateway.networking.k8s.io

CRD name and creation date

kubectl get crd gatewayclasses.gateway.networking.k8s.io -o jsonpath='{.status.storedVersions}'

Includes v1.

kubectl get crd tcproutes.gateway.networking.k8s.io

CRD exists

kubectl get gatewayclass <name> -o jsonpath='{.status.conditions[?(@.type=="Accepted")].status}'

True

kubectl get gateway <name> -n <ns> -o jsonpath='{.status.conditions[?(@.type=="Accepted")].status}'

True

dig +short test.cloudbees.example.com

Gateway external IP or hostname

kubectl get secret <name> -n <gateway-ns> -o jsonpath='{.type}'

kubernetes.io/tls

kubectl get namespace <ns> --show-labels

Label is present.

kubectl auth can-i create httproutes.gateway.networking.k8s.io --as=system:serviceaccount:<ns>:<sa> -n <ns>

yes

Next steps

After all pre-flight checks pass, choose a namespace topology for your CloudBees CI on modern cloud platforms deployment, and then install CloudBees CI on modern cloud platforms with Gateway API.