Kubernetes configuration options

How to configure CloudBees CD/RO for very large Helm deployments

A very large Helm deployment of CloudBees CD/RO is:

~ 100K jobs per day
~ 2000 running pipelines per day
~ 5M job steps per day

Active customers and partners can refer to the CloudBees CD/RO Level 2: Helm for Very Large Kubernetes Deployments training course for details.

The following tables summarize the changes to the default Helm chart for very large deployments:

CloudBees CD/RO server values

For the default CloudBees CD/RO Helm charts, refer to CloudBees CD/RO server values. You can find these cloudbees-flow values referenced in the values.yaml under the Flow server configuration section.

Table 1. CloudBees CD/RO server parameter descriptions and values for very large Helm deployments
Name	Value
server.replicas	4
server.ecconfigure	"--serverInitMemory=70 --serverMaxMemory=85"
server.resources.limits.cpu	16
server.resources.limits.memory	128Gi
server.resources.requests.cpu	4
server.resources.requests.memory	16Gi

CloudBees CD/RO web server values

For the default CloudBees CD/RO Helm charts, refer to CloudBees CD/RO web server values. You can find these cloudbees-flow values referenced in the values.yaml under the Flow web server configuration section.

Table 2. CloudBees CD/RO web server parameter descriptions and values for very large Helm deployments
Name	Value
web.replicas	2

Repository values

For the default CloudBees CD/RO Helm charts, refer to Repository values. You can find these cloudbees-flow values referenced in the values.yaml under the Flow repository configuration section.

Table 3. Repository parameter descriptions and values for very large Helm deployments
Name	Value
repository.resources.requests.memory	768Mi

CloudBees Analytics server values

For the default CloudBees CD/RO Helm charts, refer to CloudBees Analytics server values. You can find these cloudbees-flow values referenced in the values.yaml under the Analytics server configuration section.

Table 4. CloudBees Analytics server parameter descriptions and values for very large Helm deployments
Name	Value
analytics.replicas	`3`
analytics.esMinMasterNodes	`2`
analytics.heapSize	`8192`
analytics.resources.limits.cpu	`4`
analytics.resources.limits.memory	`16Gi`
analytics.resources.requests.cpu	`4`
analytics.resources.requests.memory	`16Gi`

CloudBees CD/RO agent values

For the default CloudBees CD/RO Helm charts, refer to CloudBees CD/RO agent configuration values You can find these cloudbees-flow-agent values referenced in the agent values.yaml.

Table 5. CloudBees CD/RO agent parameter descriptions and values for very large Helm deployments
Name	Value
replicas	`2`

How to set global values in CloudBees CD/RO Helm charts

Starting with v2023.10.0, you can configure global values for image settings to use for all CloudBees CD/RO components. In v2023.10.0 and later cloudbees-flow values files, you can find these configurations by searching for Global configurations section.

When you set a global image, it is applied to the following components:

server
web
repository
analytics
analytics.backup
boundAgent
zookeeper
internalGatewayAgent
externalGatewayAgent

This option improves usability and enables you to set and update chart values for these components in a single location, instead of for each component individually. However, this setting is optional, and you can still configure CloudBees CD/RO components individually to pull specific images.

The following sections do not apply values from the global section and must be individually configured:

analytics.sysctlInitContainer
analytics.backup.cleanupImageRepository
analytics.backup.cleanupImageTag
mariadb
ingress-nginx
cdsidecarinjector
helmTests

For information on setting custom images, refer to How to configure custom images for CloudBees CD/RO.

Setting global chart values

Setting global values for image settings is an optional configuration for CloudBees CD/RO v2023.10.0 and later, meant to improve usability and maintainability of your Helm charts. Global chart values allow you to set a value in one location that, where applicable, is applied to all CloudBees CD/RO components.

Global-level chart values override component-level chart values. This means, if you have both a global-level value set and component-level value set, the global value is taken.

Before setting any global-level value, CloudBees strongly suggests to ensure the global value is valid and correct for all instances where it will be applied.

The following is an example of setting a global image value:

In your v2023.10.0 and later values file, navigate to the Global configurations section.
Provide values for:
- global.cloudbees.imageRegistry
- global.cloudbees.imageTag
- (OPTIONAL) global.cloudbees.imagePullSecrets
  
  You must enable imagePullSecrets if you are using a private registry. Secrets must be manually created or already exist in the namespace.

(OPTIONAL) For CloudBees CD/RO component values that have a global.<value> set, remove the <component>.<value>.

Global-level configurations override component-level configurations. However, from a troubleshooting point of view, having two different values configured in different locations in your values file may cause some confusion. Because of this, CloudBees suggests removing the component-level configuration for values already set at the global level.

You are now able to update the global image to use for all CloudBees CD/RO components in a single location in your values file.

How to configure custom resources for init job

If using CloudBees CD/RO v2024.06.0 or later, you can configure the resources allocated to the CloudBees CD/RO init job within your values file using jobInit.resources. To get started:

If you do not already have a v2024.06.0 or later values file, update your existing values file with the following fields for jobInit:
```
  resources:
    limits:
      cpu: 4
      memory: 6Gi
    requests:
      cpu: 2
      memory: 6Gi▼
```

Set your custom values for the fields.

The default values are the minimum suggested values. For some environments these settings may not be sufficient, and result in the init job failing. If this occurs, increase the default settings for your environment, and run the installation or upgrade again.

Ensure the YAML is valid, and save your changes.
Deploy the updated chart to your environment using your helm upgrade command.

The CloudBees CD/RO init job will now be allocated the configured resources.

How to configure custom images for CloudBees CD/RO

If you need to use custom images for individual components in your project, you can configure the image information in your myvalues.yaml. The following sections describe how to set custom images for:

Bound agent
ingress-nginx
CloudBees sidecar injector
MariaDB
ZooKeeper

If you set a values.global.cloudbees image and a component-level image for bound agents in either the default CloudBees CD/RO values file or your myvalues.yaml, the values.global.cloudbees image value overrides the component-level image values.

Before configuring custom images for bound agents, ensure you do not have images configured for values.global.cloudbees within your values files.

Configure custom bound agent images

This section is intended for use with v2023.10.0 and later Helm charts. For CloudBees CD/RO Helm charts v2023.08.0 and earlier, the boundAgent included image configuration fields in the Helm chart by default.

Add a custom boundAgent image to Helm chart

In CloudBees CD/RO v2023.10.0 and above Helm charts, global image configurations were implemented in the cloudbees-flow values file, values.global.cloudbees. As part of this change, bound agents were reconfigured to use the cloudbees-flow-agent.values.image values. Additionally, the boundAgent.imageRegistry values were removed from the cloudbees-flow values file.

If your project uses custom images for bound agents, you must make the following changes to CloudBees CD/RO v2023.10.0 and above Helm charts before starting the upgrade:

In your cloudbees-flow myvalues.yaml, ensure no values.global.cloudbees image value is set.

If you set both a values.global.cloudbees image value and any component-level image values in your myvalues.yaml, the values.global.cloudbees image value overrides all component-level image values.

In the boundAgent section of your cloudbees-flow values file, add the following chart configuration:

  images:
    ## Image registry to pull CloudBees CD/RO boundAgent images from.
    ## Example: registry: "123456789012.dkr.ecr.us-east-1.amazonaws.com"
    registry: "<YOUR-IMAGE-REGISTRY>"
    ## The `imageRepository` in the `images.registry` to pull the bound agent image from.
    imageRepository: "<YOUR-IMAGE-REPO>"
    ## CloudBees CD/RO bound agent image tag to pull.
    tag: "<YOUR-IMAGE-TAG>"▼

For example, if your project uses an internal registry called internal.registry.example/cloudbees::

  images:
    ## Image registry to pull CloudBees CD/RO boundAgent images from.
    ## Example: registry: "123456789012.dkr.ecr.us-east-1.amazonaws.com"
    registry: "internal.registry.example/cloudbees"
    ## The `imageRepository` in the `images.registry` to pull the bound agent image from.
    imageRepository: "cbflow-boundagent"
    ## CloudBees CD/RO bound agent image tag to pull.
    tag: "1.0"▼

You can now upgrade to CloudBees CD/RO v2023.10.0 and above using your custom bound agent image.

Configure custom ingress-nginx images

Add a custom ingress-nginx image to Helm chart

If your project uses custom images for ingress-nginx, you must make the following image changes to the CloudBees CD/RO Helm charts in your myvalues.yaml before starting the upgrade.

In the ingress-nginx section of your cloudbees-flow values file, add the following chart configuration:

  controller:
    image:
      registry: <YOUR-IMAGE-REGISTRY>
      image: <YOUR-IMAGE>
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format. Otherwise, installing the chart by providing the `values.yaml` will fail.
      # repository:
      tag: "<YOUR-IMAGE-TAG>"
      digest: <YOUR-DIGEST-HASH>
    admissionWebhooks:
      patch:
        registry: <YOUR-IMAGE-REGISTRY>
        image: <YOUR-IMAGE>
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format. Otherwise, installing the chart by providing the `values.yaml` will fail.
        # repository:
        tag: "<YOUR-IMAGE-TAG>"
        digest: <YOUR-DIGEST-HASH>
  ## Set images if enabled
  defaultBackend:
    enabled: false
    image:
      registry: <YOUR-IMAGE-REGISTRY>
      image: <YOUR-IMAGE>
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format or installing the chart by providing the `values.yaml` will fail.
      # repository:
      tag: "<YOUR-IMAGE-TAG>"▼

For example, if your project uses an internal registry called internal.registry.example/ingress-nginx:

  controller:
    image:
      registry: internal.registry.example/ingress-nginx
      image: controller
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format or installing the chart by providing the `values.yaml` will fail.
      # repository:
      tag: "v1.1.0"
      digest: sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a
    admissionWebhooks:
      patch:
        registry: internal.registry.example/ingress-nginx
        image: kube-webhook-certgen
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format or installing the chart by providing the `values.yaml` will fail.
        # repository:
        tag: "v1.1.1"
        digest: sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660
  ## Set images if enabled
  defaultBackend:
    enabled: false
    image:
      registry: internal.registry.example/ingress-nginx
      image: defaultbackend-am