Kubernetes configuration options

How to configure CloudBees CD/RO for very large Helm deployments

A very large Helm deployment of CloudBees CD/RO is:

~ 100K jobs per day
~ 2000 running pipelines per day
~ 5M job steps per day

Active customers and partners can refer to the CloudBees CD/RO Level 2: Helm for Very Large Kubernetes Deployments training course for details.

The following tables summarize the changes to the default Helm chart for very large deployments:

CloudBees CD/RO server values

For the default CloudBees CD/RO Helm charts, refer to CloudBees CD/RO server values. You can find these cloudbees-flow values referenced in the values.yaml under the Flow server configuration section.

Table 1. CloudBees CD/RO server parameter descriptions and values for very large Helm deployments
Name	Value
server.replicas	4
server.ecconfigure	"--serverInitMemory=70 --serverMaxMemory=85"
server.resources.limits.cpu	16
server.resources.limits.memory	128Gi
server.resources.requests.cpu	4
server.resources.requests.memory	16Gi

CloudBees CD/RO web server values

For the default CloudBees CD/RO Helm charts, refer to CloudBees CD/RO web server values. You can find these cloudbees-flow values referenced in the values.yaml under the Flow web server configuration section.

Table 2. CloudBees CD/RO web server parameter descriptions and values for very large Helm deployments
Name	Value
web.replicas	2

Repository values

For the default CloudBees CD/RO Helm charts, refer to Repository values. You can find these cloudbees-flow values referenced in the values.yaml under the Flow repository configuration section.

Table 3. Repository parameter descriptions and values for very large Helm deployments
Name	Value
repository.resources.requests.memory	768Mi

CloudBees Analytics server values

For the default CloudBees CD/RO Helm charts, refer to CloudBees Analytics server values. You can find these cloudbees-flow values referenced in the values.yaml under the Analytics server configuration section.

Table 4. CloudBees Analytics server parameter descriptions and values for very large Helm deployments
Name	Value
analytics.replicas	`3`
analytics.esMinMasterNodes	`2`
analytics.heapSize	`8192`
analytics.resources.limits.cpu	`4`
analytics.resources.limits.memory	`16Gi`
analytics.resources.requests.cpu	`4`
analytics.resources.requests.memory	`16Gi`

CloudBees CD/RO agent values

For the default CloudBees CD/RO Helm charts, refer to CloudBees CD/RO agent configuration values You can find these cloudbees-flow-agent values referenced in the agent values.yaml.

Table 5. CloudBees CD/RO agent parameter descriptions and values for very large Helm deployments
Name	Value
replicas	`2`

How to set global values in CloudBees CD/RO Helm charts

Starting with v2023.10.0, you can configure global values for image settings to use for all CloudBees CD/RO components. In v2023.10.0 and later cloudbees-flow values files, you can find these configurations by searching for Global configurations section.

When you set a global image, it is applied to the following components:

server
web
repository
analytics
analytics.backup
boundAgent
zookeeper
internalGatewayAgent
externalGatewayAgent

This option improves usability and enables you to set and update chart values for these components in a single location, instead of for each component individually. However, this setting is optional, and you can still configure CloudBees CD/RO components individually to pull specific images.

The following sections do not apply values from the global section and must be individually configured:

analytics.sysctlInitContainer
analytics.backup.cleanupImageRepository
analytics.backup.cleanupImageTag
mariadb
ingress-nginx
cdsidecarinjector
helmTests

For information on setting custom images, refer to How to configure custom images for CloudBees CD/RO.

Setting global chart values

Setting global values for image settings is an optional configuration for CloudBees CD/RO v2023.10.0 and later, meant to improve usability and maintainability of your Helm charts. Global chart values allow you to set a value in one location that, where applicable, is applied to all CloudBees CD/RO components.

Global-level chart values override component-level chart values. This means, if you have both a global-level value set and component-level value set, the global value is taken.

Before setting any global-level value, CloudBees strongly suggests to ensure the global value is valid and correct for all instances where it will be applied.

The following is an example of setting a global image value:

In your v2023.10.0 and later values file, navigate to the Global configurations section.
Provide values for:
- global.cloudbees.imageRegistry
- global.cloudbees.imageTag
- (OPTIONAL) global.cloudbees.imagePullSecrets
  
  You must enable imagePullSecrets if you are using a private registry. Secrets must be manually created or already exist in the namespace.

(OPTIONAL) For CloudBees CD/RO component values that have a global.<value> set, remove the <component>.<value>.

Global-level configurations override component-level configurations. However, from a troubleshooting point of view, having two different values configured in different locations in your values file may cause some confusion. Because of this, CloudBees suggests removing the component-level configuration for values already set at the global level.

You are now able to update the global image to use for all CloudBees CD/RO components in a single location in your values file.

How to configure custom resources for init job

If using CloudBees CD/RO v2024.06.0 or later, you can configure the resources allocated to the CloudBees CD/RO init job within your values file using jobInit.resources. To get started:

If you do not already have a v2024.06.0 or later values file, update your existing values file with the following fields for jobInit:
```
  resources:
    limits:
      cpu: 4
      memory: 6Gi
    requests:
      cpu: 2
      memory: 6Gi▼
```

Set your custom values for the fields.

The default values are the minimum suggested values. For some environments these settings may not be sufficient, and result in the init job failing. If this occurs, increase the default settings for your environment, and run the installation or upgrade again.

Ensure the YAML is valid, and save your changes.
Deploy the updated chart to your environment using your helm upgrade command.

The CloudBees CD/RO init job will now be allocated the configured resources.

How to configure custom images for CloudBees CD/RO

If you need to use custom images for individual components in your project, you can configure the image information in your myvalues.yaml. The following sections describe how to set custom images for:

Bound agent
ingress-nginx
CloudBees sidecar injector
MariaDB
ZooKeeper

If you set a values.global.cloudbees image and a component-level image for bound agents in either the default CloudBees CD/RO values file or your myvalues.yaml, the values.global.cloudbees image value overrides the component-level image values.

Before configuring custom images for bound agents, ensure you do not have images configured for values.global.cloudbees within your values files.

Configure custom bound agent images

This section is intended for use with v2023.10.0 and later Helm charts. For CloudBees CD/RO Helm charts v2023.08.0 and earlier, the boundAgent included image configuration fields in the Helm chart by default.

Add a custom boundAgent image to Helm chart

In CloudBees CD/RO v2023.10.0 and above Helm charts, global image configurations were implemented in the cloudbees-flow values file, values.global.cloudbees. As part of this change, bound agents were reconfigured to use the cloudbees-flow-agent.values.image values. Additionally, the boundAgent.imageRegistry values were removed from the cloudbees-flow values file.

If your project uses custom images for bound agents, you must make the following changes to CloudBees CD/RO v2023.10.0 and above Helm charts before starting the upgrade:

In your cloudbees-flow myvalues.yaml, ensure no values.global.cloudbees image value is set.

If you set both a values.global.cloudbees image value and any component-level image values in your myvalues.yaml, the values.global.cloudbees image value overrides all component-level image values.

In the boundAgent section of your cloudbees-flow values file, add the following chart configuration:

  images:
    ## Image registry to pull CloudBees CD/RO boundAgent images from.
    ## Example: registry: "123456789012.dkr.ecr.us-east-1.amazonaws.com"
    registry: "<YOUR-IMAGE-REGISTRY>"
    ## The `imageRepository` in the `images.registry` to pull the bound agent image from.
    imageRepository: "<YOUR-IMAGE-REPO>"
    ## CloudBees CD/RO bound agent image tag to pull.
    tag: "<YOUR-IMAGE-TAG>"▼

For example, if your project uses an internal registry called internal.registry.example/cloudbees::

  images:
    ## Image registry to pull CloudBees CD/RO boundAgent images from.
    ## Example: registry: "123456789012.dkr.ecr.us-east-1.amazonaws.com"
    registry: "internal.registry.example/cloudbees"
    ## The `imageRepository` in the `images.registry` to pull the bound agent image from.
    imageRepository: "cbflow-boundagent"
    ## CloudBees CD/RO bound agent image tag to pull.
    tag: "1.0"▼

You can now upgrade to CloudBees CD/RO v2023.10.0 and above using your custom bound agent image.

Configure custom ingress-nginx images

Add a custom ingress-nginx image to Helm chart

If your project uses custom images for ingress-nginx, you must make the following image changes to the CloudBees CD/RO Helm charts in your myvalues.yaml before starting the upgrade.

In the ingress-nginx section of your cloudbees-flow values file, add the following chart configuration:

  controller:
    image:
      registry: <YOUR-IMAGE-REGISTRY>
      image: <YOUR-IMAGE>
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format. Otherwise, installing the chart by providing the `values.yaml` will fail.
      # repository:
      tag: "<YOUR-IMAGE-TAG>"
      digest: <YOUR-DIGEST-HASH>
    admissionWebhooks:
      patch:
        registry: <YOUR-IMAGE-REGISTRY>
        image: <YOUR-IMAGE>
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format. Otherwise, installing the chart by providing the `values.yaml` will fail.
        # repository:
        tag: "<YOUR-IMAGE-TAG>"
        digest: <YOUR-DIGEST-HASH>
  ## Set images if enabled
  defaultBackend:
    enabled: false
    image:
      registry: <YOUR-IMAGE-REGISTRY>
      image: <YOUR-IMAGE>
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format or installing the chart by providing the `values.yaml` will fail.
      # repository:
      tag: "<YOUR-IMAGE-TAG>"▼

For example, if your project uses an internal registry called internal.registry.example/ingress-nginx:

  controller:
    image:
      registry: internal.registry.example/ingress-nginx
      image: controller
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format or installing the chart by providing the `values.yaml` will fail.
      # repository:
      tag: "v1.1.0"
      digest: sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a
    admissionWebhooks:
      patch:
        registry: internal.registry.example/ingress-nginx
        image: kube-webhook-certgen
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format or installing the chart by providing the `values.yaml` will fail.
        # repository:
        tag: "v1.1.1"
        digest: sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660
  ## Set images if enabled
  defaultBackend:
    enabled: false
    image:
      registry: internal.registry.example/ingress-nginx
      image: defaultbackend-amd64
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format or installing the chart by providing the `values.yaml` will fail.
      # repository:
      tag: "1.5"▼

You can now upgrade CloudBees CD/RO to use your custom ingress-nginx image.

Configure custom CloudBees sidecar injector images

Add a custom CloudBees sidecar injector image to Helm chart

If your project uses custom images for a CloudBees sidecar injector, you must make the following images changes to the CloudBees CD/RO Helm charts in your myvalues.yaml before starting the upgrade.

In the cloudbees sidecar injector (cdsidecarinjector) section of your cloudbees-flow values file, add the following chart configuration:

  enabled=true
  hub=<YOUR-IMAGE-REGISTRY>
  image=<YOUR-IMAGE>
  tag="<YOUR-IMAGE-TAG>"
  requestCert:
    image=<YOUR-IMAGE>
    tag="<YOUR-IMAGE-TAG>"▼

For example, if your project uses an internal registry called internal.registry.example/cloudbees:

  enabled=true
  hub=internal.registry.example/cloudbees
  image=test-sidecar-injector
  tag="2.2.1"
  requestCert:
    image=test-cert-requester
    tag="2.2.1"▼

You can now upgrade CloudBees CD/RO to use your custom cloudbees-sidecar-injector image.

Configure custom MariaDB images

Add a custom MariaDB image to Helm chart

If your project uses custom images for MariaDB, you must make the following images changes to the CloudBees CD/RO Helm charts in your myvalues.yaml before starting the upgrade.

In the mariadb section of your cloudbees-flow values file, add the following chart configuration:

  image:
    registry: <YOUR-IMAGE-REGISTRY>
    repository: <YOUR-IMAGE-REPO>
    tag: <YOUR-IMAGE-TAG>▼

For example, if your project uses an internal registry called internal.registry.example/cloudbees:

  image:
    registry: internal.registry.example/cloudbees
    repository: mariadb
    tag: 10.3.20-debian-9-r0▼

You can now upgrade CloudBees CD/RO to use your custom MariaDB image.

Configure custom ZooKeeper images

Add a custom ZooKeeper image to Helm chart with a global image set

If your project uses custom images for ZooKeeper, and has global values set, you must make the following images changes to the CloudBees CD/RO Helm charts in your myvalues.yaml before starting the upgrade:

In your cloudbees-flow myvalues.yaml, ensure the values.global.cloudbees.imageRegistry and values.global.cloudbees.imageTag image values are set.

Ensure your ZooKeeper image is located in the values.global.cloudbees.imageRegistry, and it set with the values.global.cloudbees.imageTag. If it is not, your ZooKeeper image will fail to load. If your project is not using these values, refer to Add a custom ZooKeeper image to Helm chart with no global image set.

In the zookeeper section of your cloudbees-flow values file, add the following chart configuration:
```
  image:
    imageRepository: <YOUR-IMAGE-REPO>▼
```
- For example, if your project uses a global registry called global-images-registry.example with image tags set to global-image-tag, and your ZooKeeper image was in the zookeeper-image repository, your values file should have the following configurations:
  global: cloudbees: imageRegistry: global-images-registry.example imageTag: global-image-tag .... zookeeper: image: imageRepository: zookeeper-image
  ▼

You can now upgrade CloudBees CD/RO to use your custom ZooKeeper image.

Add a custom ZooKeeper image to Helm chart with no global image set

If your project uses custom images for ZooKeeper, and has no global image values set, you must make the following images changes to the CloudBees CD/RO Helm charts in your myvalues.yaml before starting the upgrade:

In your cloudbees-flow myvalues.yaml, ensure no values.global.cloudbees image value is set.

If you have values.global.cloudbees.imageRegistry and values.global.cloudbees.imageTag, your ZooKeeper image will fail to load using the following configuration. If you are using these values, refer to Add a custom ZooKeeper image to Helm chart with a global image set.

In the zookeeper section of your cloudbees-flow values file, add the following chart configuration:

  image:
    repository: <YOUR-IMAGE-REGISTRY>/<YOUR-IMAGE-REPO>
    tag: "<YOUR-IMAGE-TAG>"▼

For example, if your project uses an internal registry called internal.registry.example/ with image repository called zookeeper-image:

 image:
    ## Container repository to pull ZooKeeper image from.
    repository: internal.registry.example/zookeeper-image
    ## Zookeeper image tag to pull.
    tag: "2023.08.0.167214_3.2.51_20230809"▼

You can now upgrade CloudBees CD/RO to use your custom ZooKeeper image.

How to configure load balancing with ingress

For the default CloudBees CD/RO Helm charts, refer to Ingress values. You can find these cloudbees-flow values referenced in the values.yaml under the Flow ingress configuration section.

An ingress controller typically does not eliminate the need for an external load balancer but adds a layer of routing and control behind the load balancer. However, you can configure the ingress controller so all service endpoints, such as web, server, and repository, may be exposed from the same domain name and load balancer endpoint.

CloudBees recommends configuring your ingress controller so all CloudBees CD/RO services can be exposed through a single load balancer.

By default, ingress is enabled in the CloudBees CD/RO chart. The following is a summary of the settings:

To run CloudBees CD/RO with Kubernetes 1.22 and later, you must use the ingress-nginx controller with the following required settings:

ingress-nginx.enabled=true
ingress.class=nginx
nginx-ingress.enabled=false

Kubernetes versions 1.21 and earlier

nginx-ingress.controller.ingressClass

Default: flow-ingress

nginx-ingress.controller.publishService.enabled

Default: true

nginx-ingress.controller.scope.enabled

Default: true

nginx-ingress.enabled

Default: true

nginx-ingress.tcp.61613

CloudBees CD/RO server

Default: {{ .Release.Namespace }}/flow-server:61613

nginx-ingress.tcp.8200

CloudBees CD/RO repository

Default: {{ .Release.Namespace }}/flow-repository:8200

nginx-ingress.tcp.8443

CloudBees CD/RO web server

Default: {{ .Release.Namespace }}/flow-server:8443

nginx-ingress.tcp.9201

CloudBees Analytics database

Default: {{ .Release.Namespace }}/flow-analytics:9201

Kubernetes versions 1.22 and later

ingress-nginx.controller.ingressClass

Default: flow-ingress

ingress-nginx.controller.publishService.enabled

Default: true

ingress-nginx.controller.scope.enabled

Default: true

ingress-nginx.enabled

Default: true

ingress-nginx.tcp.61613

CloudBees CD/RO server

Default: {{ .Release.Namespace }}/flow-server:61613

ingress-nginx.tcp.8200

CloudBees CD/RO repository

Default: {{ .Release.Namespace }}/flow-repository:8200

ingress-nginx.tcp.8443

CloudBees CD/RO web server

Default: {{ .Release.Namespace }}/flow-server:8443

ingress-nginx.tcp.9201

CloudBees Analytics database

Default: {{ .Release.Namespace }}/flow-analytics:9201

How to install CloudBees CD/RO on Kubernetes using an existing database

You can install CloudBees CD/RO using your existing database, if you are:

Performing an initial installation of CloudBees CD/RO.
Migrating from a traditional CloudBees CD/RO installation to Kubernetes.
Spinning up an additional testing or production environment.
Upgrading between CloudBees CD/RO versions.

The procedures to install your existing database with CloudBees CD/RO are nearly the same for each of these scenarios. However, if this is your initial installation, it may be easier to install CloudBees CD/RO using the instructions found in Install CloudBees CD/RO production server, and then update your installation with the steps listed here.

If you only want to find the values needed to configure your existing database as part of the Helm chart customization process, you can also refer to Database values.

Before you start

Ensure the CloudBees CD/RO version you are installing or upgrading to supports the database you want to integrate. For more information on which databases are supported by each CloudBees CD/RO release, refer to Supported platforms for CloudBees CD/RO on Kubernetes.
Always make backups of your project-specific resources prior to installing or upgrading your CloudBees CD/RO version. These backups may be vital to recovering your data if critical errors occur during upgrades.
Always perform adequate and thorough testing in a development environment before releasing any version installation or upgrade into your production environment.
To see an example of a complete CloudBees CD/RO installation (not upgrade) using an existing database, refer to Example installation of CloudBees CD/RO using an existing database.

If you do not already have a myvalues.yaml, refer to CloudBees CD/RO default Helm charts to set yours up before continuing these steps.

Installing CloudBees CD/RO with your existing database

To install your existing database:

Make backups of your deployment and database. These backups may be necessary to recover data if critical errors occurs during the upgrade process.

Set the variables for the current release’s name, its namespace, and version you want to install/upgrade to:

releaseName="<your-current-release>"
namespace="<your-current-release-namespace>"

# Example upgradeVersion value: "2.21.0"
upgradeVersion="<version-of-upgrade-release>"▼

Save the values from your current release to a values file by running:
```
helm get values $releaseName --namespace $namespace > old-values.yaml▼
```
If this is your initial installation, this step is not needed, you can continue and use the same steps for your myvalues.yaml as described for old-values.yaml.
Ensure your old-values.yaml contains values, or set the values, for the following parameters:
- database.externalEndpoint
- database.dbPort
- database.dbType
- database.dbName
- database.dbUser
- database.dbPassword
- flowLicense
- flowCredentials.adminPassword
  
  When using the helm install command, you can use the --set option to specify these values.

If your old-values.yaml uses the images.tag, you must update it with the image tag of the new CloudBees CD/RO release you want to upgrade to.

If this is your initial installation, and you downloaded the latest version of the values file from SonaType Nexus, your values file already contains this information unless you removed it during customization.

If you do not update the images.tag or update it with an incorrect value, your installation/upgrade will not succeed.

After you have performed these steps, set any additional values you need in your old-values.yaml for the CloudBees CD/RO installation. For more information refer to Configure Helm charts.

Once you have your values file ready for installation, ensure the network connection between your database instance and Kubernetes cluster is active before continuing.

Next, you need your flow-server passkey and keystore files from your existing CloudBees CD/RO installation and to convert them to base64 format:

Copy and save your flow-server passkey:

# Get the flow-server pod name
cdServerPod=$(kubectl get pod -l app=flow-server -o \
jsonpath='{.items[*].metadata.name}' -n $namespace); echo $cdServerPod

# Get the passkey file name
passkey=$(kubectl -n $namespace exec $cdServerPod -- ls /tmp/ | grep passkey); echo $passkey

# Copy the passkey file from the flow-server pod to the local machine
kubectl -n $namespace cp $cdServerPod:/tmp/$passkey ./passkey

# Ensure the passkey file has been copied
ls -l | grep passkey▼

Convert your passkey to base64 format:
```
base64 -i passkey -o passkey.b64
```