Kubernetes configuration options

How to configure CloudBees CD/RO for very large Helm deployments

A very large Helm deployment of CloudBees CD/RO is:

~ 100K jobs per day
~ 2000 running pipelines per day
~ 5M job steps per day

Active customers and partners can refer to the CloudBees CD/RO Level 2: Helm for Very Large Kubernetes Deployments training course for details.

The following tables summarize the changes to the default Helm chart for very large deployments:

CloudBees CD/RO server values

For the default CloudBees CD/RO Helm charts, refer to CloudBees CD/RO server values. You can find these cloudbees-flow values referenced in the values.yaml under the Flow server configuration section.

Table 1. CloudBees CD/RO server parameter descriptions and values for very large Helm deployments
Name	Value
server.replicas	4
server.ecconfigure	"--serverInitMemory=70 --serverMaxMemory=85"
server.resources.limits.cpu	16
server.resources.limits.memory	128Gi
server.resources.requests.cpu	4
server.resources.requests.memory	16Gi

CloudBees CD/RO web server values

For the default CloudBees CD/RO Helm charts, refer to CloudBees CD/RO web server values. You can find these cloudbees-flow values referenced in the values.yaml under the Flow web server configuration section.

Table 2. CloudBees CD/RO web server parameter descriptions and values for very large Helm deployments
Name	Value
web.replicas	2

Repository values

For the default CloudBees CD/RO Helm charts, refer to Repository values. You can find these cloudbees-flow values referenced in the values.yaml under the Flow repository configuration section.

Table 3. Repository parameter descriptions and values for very large Helm deployments
Name	Value
repository.resources.requests.memory	768Mi

CloudBees Analytics server values

For the default CloudBees CD/RO Helm charts, refer to CloudBees Analytics server values. You can find these cloudbees-flow values referenced in the values.yaml under the Flow DevOps Insight configuration section.

Table 4. CloudBees Analytics server parameter descriptions and values for very large Helm deployments
Name	Value
dois.replicas	`3`
dois.esMinMasterNodes	`2`
dois.esRam	`8192`
dois.lsMaxRam	`2048`
dois.resources.limits.cpu	`4`
dois.resources.limits.memory	`16Gi`
dois.resources.requests.cpu	`4`
dois.resources.requests.memory	`16Gi`

CloudBees CD/RO agent values

For the default CloudBees CD/RO Helm charts, refer to CloudBees CD/RO agent configuration values You can find these cloudbees-flow-agent values referenced in the agent values.yaml.

Table 5. CloudBees CD/RO agent parameter descriptions and values for very large Helm deployments
Name	Value
replicas	`2`

How to set global values in CloudBees CD/RO Helm charts

Starting with v2023.10.0, you can configure global values for image settings to use for all CloudBees CD/RO components. In v2023.10.0 and later cloudbees-flow values files, you can find these configurations by searching for Global configurations section.

When you set a global image, it is applied to the following components:

server
web
repository
dois
dois.backup
boundAgent
zookeeper
internalGatewayAgent
externalGatewayAgent

This option improves usability and enables you to set and update chart values for these components in a single location, instead of for each component individually. However, this setting is optional, and you can still configure CloudBees CD/RO components individually to pull specific images.

The following sections do not apply values from the global section and must be individually configured:

dois.sysctlInitContainer
dois.backup.cleanupImageRepository
dois.backup.cleanupImageTag
mariadb
ingress-nginx
cdsidecarinjector
helmTests

For information on setting custom images, refer to How to configure custom images for CloudBees CD/RO.

Setting global chart values

Setting global values for image settings is an optional configuration for CloudBees CD/RO v2023.10.0 and later, meant to improve usability and maintainability of your Helm charts. Global chart values allow you to set a value in one location that, where applicable, is applied to all CloudBees CD/RO components.

Global-level chart values override component-level chart values. This means, if you have both a global-level value set and component-level value set, the global value is taken.

Before setting any global-level value, CloudBees strongly suggests to ensure the global value is valid and correct for all instances where it will be applied.

The following is an example of setting a global image value:

In your v2023.10.0 and later values file, navigate to the Global configurations section.
Provide values for:
- global.cloudbees.imageRegistry
- global.cloudbees.imageTag
- (OPTIONAL) global.cloudbees.imagePullSecrets
  
  You must enable imagePullSecrets if you are using a private registry. Secrets must be manually created or already exist in the namespace.

(OPTIONAL) For CloudBees CD/RO component values that have a global.<value> set, remove the <component>.<value>.

Global-level configurations override component-level configurations. However, from a troubleshooting point of view, having two different values configured in different locations in your values file may cause some confusion. Because of this, CloudBees suggests removing the component-level configuration for values already set at the global level.

You are now able to update the global image to use for all CloudBees CD/RO components in a single location in your values file.

How to configure custom images for CloudBees CD/RO

If you need to use custom images for individual components in your project, you can configure the image information in your myvalues.yaml. The following sections describe how to set custom images for:

Bound agent
ingress-nginx
CloudBees sidecar injector
MariaDB
ZooKeeper

If you set a values.global.cloudbees image and a component-level image for bound agents in either the default CloudBees CD/RO values file or your myvalues.yaml, the values.global.cloudbees image value overrides the component-level image values.

Before configuring custom images for bound agents, ensure you do not have images configured for values.global.cloudbees within your values files.

Configure custom bound agent images

This section is intended for use with v2023.10.0 and later Helm charts. For CloudBees CD/RO Helm charts v2023.08.0 and earlier, the boundAgent included image configuration fields in the Helm chart by default.

Add a custom boundAgent image to Helm chart

In CloudBees CD/RO v2023.10.0 and above Helm charts, global image configurations were implemented in the cloudbees-flow values file, values.global.cloudbees. As part of this change, bound agents were reconfigured to use the cloudbees-flow-agent.values.image values. Additionally, the boundAgent.imageRegistry values were removed from the cloudbees-flow values file.

If your project uses custom images for bound agents, you must make the following changes to CloudBees CD/RO v2023.10.0 and above Helm charts before starting the upgrade:

In your cloudbees-flow myvalues.yaml, ensure no values.global.cloudbees image value is set.

If you set both a values.global.cloudbees image value and any component-level image values in your myvalues.yaml, the values.global.cloudbees image value overrides all component-level image values.

In the boundAgent section of your cloudbees-flow values file, add the following chart configuration:

  images:
    ## Image registry to pull CloudBees CD/RO boundAgent images from.
    ## Example: registry: "123456789012.dkr.ecr.us-east-1.amazonaws.com"
    registry: "<YOUR-IMAGE-REGISTRY>"
    ## The `imageRepository` in the `images.registry` to pull the bound agent image from.
    imageRepository: "<YOUR-IMAGE-REPO>"
    ## CloudBees CD/RO bound agent image tag to pull.
    tag: "<YOUR-IMAGE-TAG>"▼

For example, if your project uses an internal registry called internal.registry.example/cloudbees::

  images:
    ## Image registry to pull CloudBees CD/RO boundAgent images from.
    ## Example: registry: "123456789012.dkr.ecr.us-east-1.amazonaws.com"
    registry: "internal.registry.example/cloudbees"
    ## The `imageRepository` in the `images.registry` to pull the bound agent image from.
    imageRepository: "cbflow-boundagent"
    ## CloudBees CD/RO bound agent image tag to pull.
    tag: "1.0"▼

You can now upgrade to CloudBees CD/RO v2023.10.0 and above using your custom bound agent image.

Configure custom ingress-nginx images

Add a custom ingress-nginx image to Helm chart

If your project uses custom images for ingress-nginx, you must make the following image changes to the CloudBees CD/RO Helm charts in your myvalues.yaml before starting the upgrade.

In the ingress-nginx section of your cloudbees-flow values file, add the following chart configuration:

  controller:
    image:
      registry: <YOUR-IMAGE-REGISTRY>
      image: <YOUR-IMAGE>
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format. Otherwise, installing the chart by providing the `values.yaml` will fail.
      # repository:
      tag: "<YOUR-IMAGE-TAG>"
      digest: <YOUR-DIGEST-HASH>
    admissionWebhooks:
      patch:
        registry: <YOUR-IMAGE-REGISTRY>
        image: <YOUR-IMAGE>
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format. Otherwise, installing the chart by providing the `values.yaml` will fail.
        # repository:
        tag: "<YOUR-IMAGE-TAG>"
        digest: <YOUR-DIGEST-HASH>
  ## Set images if enabled
  defaultBackend:
    enabled: false
    image:
      registry: <YOUR-IMAGE-REGISTRY>
      image: <YOUR-IMAGE>
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format or installing the chart by providing the `values.yaml` will fail.
      # repository:
      tag: "<YOUR-IMAGE-TAG>"▼

For example, if your project uses an internal registry called internal.registry.example/ingress-nginx:

  controller:
    image:
      registry: internal.registry.example/ingress-nginx
      image: controller
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format or installing the chart by providing the `values.yaml` will fail.
      # repository:
      tag: "v1.1.0"
      digest: sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a
    admissionWebhooks:
      patch:
        registry: internal.registry.example/ingress-nginx
        image: kube-webhook-certgen
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format or installing the chart by providing the `values.yaml` will fail.
        # repository:
        tag: "v1.1.1"
        digest: sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660
  ## Set images if enabled
  defaultBackend:
    enabled: false
    image:
      registry: internal.registry.example/ingress-nginx
      image: defaultbackend-amd64
      ## For backward compatibility, consider setting the full image URL via the repository value below.
      ## Use *either* the current default registry/image or the repository format or installing the chart by providing the `values.yaml` will fail.
      # repository:
      tag: "1.5"▼

You can now upgrade CloudBees CD/RO to use your custom ingress-nginx image.

Configure custom CloudBees sidecar injector images

Add a custom CloudBees sidecar injector image to Helm chart

If your project uses custom images for a CloudBees sidecar injector, you must make the following images changes to the CloudBees CD/RO Helm charts in your myvalues.yaml before starting the upgrade.

In the cloudbees sidecar injector (cdsidecarinjector) section of your cloudbees-flow values file, add the following chart configuration:

  enabled=true
  hub=<YOUR-IMAGE-REGISTRY>
  image=<YOUR-IMAGE>
  tag="<YOUR-IMAGE-TAG>"
  requestCert:
    image=<YOUR-IMAGE>
    tag="<YOUR-IMAGE-TAG>"▼

For example, if your project uses an internal registry called internal.registry.example/cloudbees:

  enabled=true
  hub=internal.registry.example/cloudbees
  image=test-sidecar-injector
  tag="2.2.1"
  requestCert:
    image=test-cert-requester
    tag="2.2.1"▼

You can now upgrade CloudBees CD/RO to use your custom cloudbees-sidecar-injector image.

Configure custom MariaDB images

Add a custom MariaDB image to Helm chart

If your project uses custom images for MariaDB, you must make the following images changes to the CloudBees CD/RO Helm charts in your myvalues.yaml before starting the upgrade.

In the mariadb section of your cloudbees-flow values file, add the following chart configuration:

  image:
    registry: <YOUR-IMAGE-REGISTRY>
    repository: <YOUR-IMAGE-REPO>
    tag: <YOUR-IMAGE-TAG>▼

For example, if your project uses an internal registry called internal.registry.example/cloudbees:

  image:
    registry: internal.registry.example/cloudbees
    repository: mariadb
    tag: 10.3.20-debian-9-r0▼

You can now upgrade CloudBees CD/RO to use your custom MariaDB image.

Configure custom ZooKeeper images

Add a custom ZooKeeper image to Helm chart with a global image set

If your project uses custom images for ZooKeeper, and has global values set, you must make the following images changes to the CloudBees CD/RO Helm charts in your myvalues.yaml before starting the upgrade:

In your cloudbees-flow myvalues.yaml, ensure the values.global.cloudbees.imageRegistry and values.global.cloudbees.imageTag image values are set.

Ensure your ZooKeeper image is located in the values.global.cloudbees.imageRegistry, and it set with the values.global.cloudbees.imageTag. If it is not, your ZooKeeper image will fail to load. If your project is not using these values, refer to Add a custom ZooKeeper image to Helm chart with no global image set.

In the zookeeper section of your cloudbees-flow values file, add the following chart configuration:
```
  image:
    imageRepository: <YOUR-IMAGE-REPO>▼
```
- For example, if your project uses a global registry called global-images-registry.example with image tags set to global-image-tag, and your ZooKeeper image was in the zookeeper-image repository, your values file should have the following configurations:
  global: cloudbees: imageRegistry: global-images-registry.example imageTag: global-image-tag .... zookeeper: image: imageRepository: zookeeper-image
  ▼

You can now upgrade CloudBees CD/RO to use your custom ZooKeeper image.

Add a custom ZooKeeper image to Helm chart with no global image set

If your project uses custom images for ZooKeeper, and has no global image values set, you must make the following images changes to the CloudBees CD/RO Helm charts in your myvalues.yaml before starting the upgrade:

In your cloudbees-flow myvalues.yaml, ensure no values.global.cloudbees image value is set.

If you have values.global.cloudbees.imageRegistry and values.global.cloudbees.imageTag, your ZooKeeper image will fail to load using the following configuration. If you are using these values, refer to Add a custom ZooKeeper image to Helm chart with a global image set.

In the zookeeper section of your cloudbees-flow values file, add the following chart configuration:

  image:
    repository: <YOUR-IMAGE-REGISTRY>/<YOUR-IMAGE-REPO>
    tag: "<YOUR-IMAGE-TAG>"▼

For example, if your project uses an internal registry called internal.registry.example/ with image repository called zookeeper-image:

 image:
    ## Container repository to pull ZooKeeper image from.
    repository: internal.registry.example/zookeeper-image
    ## Zookeeper image tag to pull.
    tag: "2023.08.0.167214_3.2.51_20230809"▼

You can now upgrade CloudBees CD/RO to use your custom ZooKeeper image.

How to configure load balancing with ingress

For the default CloudBees CD/RO Helm charts, refer to Ingress values. You can find these cloudbees-flow values referenced in the values.yaml under the Flow ingress configuration section.

An ingress controller typically does not eliminate the need for an external load balancer but adds a layer of routing and control behind the load balancer. However, you can configure the ingress controller so all service endpoints, such as web, server, and repository, may be exposed from the same domain name and load balancer endpoint.

CloudBees recommends configuring your ingress controller so all CloudBees CD/RO services can be exposed through a single load balancer.

By default, ingress is enabled in the CloudBees CD/RO chart. The following is a summary of the settings:

To run CloudBees CD/RO with Kubernetes 1.22 and later, you must use the ingress-nginx controller with the following required settings:

ingress-nginx.enabled=true
ingress.class=nginx
nginx-ingress.enabled=false

Kubernetes versions 1.21 and earlier

nginx-ingress.controller.ingressClass

Default: flow-ingress

nginx-ingress.controller.publishService.enabled

Default: true

nginx-ingress.controller.scope.enabled

Default: true

nginx-ingress.enabled

Default: true

nginx-ingress.tcp.61613

CloudBees CD/RO server

Default: {{ .Release.Namespace }}/flow-server:61613

nginx-ingress.tcp.8200

CloudBees CD/RO repository

Default: {{ .Release.Namespace }}/flow-repository:8200

nginx-ingress.tcp.8443

CloudBees CD/RO web server

Default: {{ .Release.Namespace }}/flow-server:8443

nginx-ingress.tcp.9200

CloudBees Analytics Elasticsearch database

Default: {{ .Release.Namespace }}/flow-devopsinsight:9200

nginx-ingress.tcp.9500

CloudBees Analytics server

Default: {{ .Release.Namespace }}/flow-devopsinsight:9500

Kubernetes versions 1.22 and later

ingress-nginx.controller.ingressClass

Default: flow-ingress

ingress-nginx.controller.publishService.enabled

Default: true

ingress-nginx.controller.scope.enabled

Default: true

ingress-nginx.enabled

Default: true

ingress-nginx.tcp.61613

CloudBees CD/RO server

Default: {{ .Release.Namespace }}/flow-server:61613

ingress-nginx.tcp.8200

CloudBees CD/RO repository

Default: {{ .Release.Namespace }}/flow-repository:8200

ingress-nginx.tcp.8443

CloudBees CD/RO web server

Default: {{ .Release.Namespace }}/flow-server:8443

ingress-nginx.tcp.9200

CloudBees Analytics Elasticsearch database

Default: {{ .Release.Namespace }}/flow-devopsinsight:9200

ingress-nginx.tcp.9500

CloudBees Analytics server

Default: {{ .Release.Namespace }}/flow-devopsinsight:9500

How to install CloudBees CD/RO on Kubernetes using an existing database

You can install CloudBees CD/RO using your existing database, if you are:

Performing an initial installation of CloudBees CD/RO.
Migrating from a traditional CloudBees CD/RO installation to Kubernetes.
Spinning up an additional testing or production environment.
Upgrading between CloudBees CD/RO versions.

The procedures to install your existing database with CloudBees CD/RO are nearly the same for each of these scenarios. However, if this is your initial installation, it may be easier to install CloudBees CD/RO using the instructions found in Install CloudBees CD/RO production server, and then update your installation with the steps listed here.

If you only want to find the values needed to configure your existing database as part of the Helm chart customization process, you can also refer to Database values.

Before you start

Ensure the CloudBees CD/RO version you are installing or upgrading to supports the database you want to integrate. For more information on which databases are supported by each CloudBees CD/RO release, refer to Supported platforms for CloudBees CD/RO on Kubernetes.
Always make backups of your project-specific resources prior to installing or upgrading your CloudBees CD/RO version. These backups may be vital to recovering your data if critical errors occur during upgrades.
Always perform adequate and thorough testing in a development environment before releasing any version installation or upgrade into your production environment.
To see an example of a complete CloudBees CD/RO installation (not upgrade) using an existing database, refer to Example installation of CloudBees CD/RO using an existing database.

If you do not already have a myvalues.yaml, refer to CloudBees CD/RO default Helm charts to set yours up before continuing these steps.

Installing CloudBees CD/RO with your existing database

To install your existing database:

Make backups of your deployment and database. These backups may be necessary to recover data if critical errors occurs during the upgrade process.

Set the variables for the current release’s name, its namespace, and version you want to install/upgrade to:

releaseName="<your-current-release>"
namespace="<your-current-release-namespace>"

# Example upgradeVersion value: "2.21.0"
upgradeVersion="<version-of-upgrade-release>"▼

Save the values from your current release to a values file by running:
```
helm get values $releaseName --namespace $namespace > old-values.yaml▼
```
If this is your initial installation, this step is not needed, you can continue and use the same steps for your myvalues.yaml as described for old-values.yaml.
Ensure your old-values.yaml contains values, or set the values, for the following parameters:
- database.externalEndpoint
- database.dbPort
- database.dbType
- database.dbName
- database.dbUser
- database.dbPassword
- flowLicense
- flowCredentials.adminPassword
  
  When using the helm install command, you can use the --set option to specify these values.

If your old-values.yaml uses the images.tag, you must update it with the image tag of the new CloudBees CD/RO release you want to upgrade to.

If this is your initial installation, and you downloaded the latest version of the values file from SonaType Nexus, your values file already contains this information unless you removed it during customization.

If you do not update the images.tag or update it with an incorrect value, your installation/upgrade will not succeed.

After you have performed these steps, set any additional values you need in your old-values.yaml for the CloudBees CD/RO installation. For more information refer to Configure Helm charts.

Once you have your values file ready for installation, ensure the network connection between your database instance and Kubernetes cluster is active before continuing.

Next, you need your flow-server passkey and keystore files from your existing CloudBees CD/RO installation and to convert them to base64 format:

Copy and save your flow-server passkey:

# Get the flow-server pod name
cdServerPod=$(kubectl get pod -l app=flow-server -o \
jsonpath='{.items[*].metadata.name}' -n $namespace); echo $cdServerPod

# Get the passkey file name
passkey=$(kubectl -n $namespace exec $cdServerPod -- ls /tmp/ | grep passkey); echo $passkey

# Copy the passkey file from the flow-server pod to the local machine
kubectl -n $namespace cp $cdServerPod:/tmp/$passkey ./passkey

# Ensure the passkey file has been copied
ls -l | grep passkey▼

Convert your passkey to base64 format:
```
base64 -i passkey -o passkey.b64▼
```

Copy and store the flow-server keystore files:

# Get the keystore file name
keystore=$(kubectl -n $namespace exec $cdServerPod -- ls /tmp/ | grep keystore); echo $keystore

# Copy the keystore file from flow-server pod to your local machine
kubectl -n $namespace cp $cdServerPod:/tmp/$keystore ./keystore

# Ensure the keystore file has been copied
ls -l | grep keystore▼

Convert your keystore to base64 format:
```
base64 -i keystore -o keystore.b64▼
```

Install CloudBees CD/RO using your old-values.yaml, passkey, keystore, and license.xml (if required):

helm install $releaseName cloudbees/cloudbees-flow --namespace $namespace \
--values old-values.yaml \
--set-file server.customConfig.passkey\\.b64=passkey.b64 \
--set-file server.customConfig.keystore\\.b64=keystore.b64 \
# Helm upgrades exit when all tasks are complete, regardless of timeout.
# However, if your database is very large, the timeout may need to be increased to prevent returning a FAILED status.
# For instance: --timeout 10800s
--timeout 4200s \
# If required include your license.xml
--set-file flowLicense.licenseData=license.xml▼

If you are installing a CloudBees CD/RO instance with an external database, do not use the --wait option. The flow-server-init-job cannot be started if --wait is used.

Example installation of CloudBees CD/RO using an existing database

For your reference, the following shows a complete installation example of CloudBees CD/RO using an existing database:

Example CloudBees CD/RO installation using existing database

dbType=sqlserver
dbEndpoint=mssql.k8s.xyz
dbPort=1433
dbUser=dbadmin
dbPass=mySecretDbPassword
dbName=myexistingdb
flowPassword=mySecretFlowPassword
doisPassword=mySecretDoisPassword
licenseFile=path/to/license.xml
passkeyFile=path/to/passkey
keystoreFile=path/to/keystore
storageClass=nfs-client
hostname=flow-server.k8s.xyz
namespace=test-existing-db
release=cloudbees-flow
releaseVersion=2.21

base64 -i $passkeyFile -o passkey.b64
base64 -i $keystoreFile -o keystore.b64

kubectl create namespace $namespace

kubectl -n $namespace create secret \
generic $release-cloudbees-flow-credentials \
--from-literal CBF_SERVER_ADMIN_PASSWORD=$flowPassword

kubectl -n $namespace create secret \
generic $release-cloudbees-flow-dois \
--from-literal CBF_DOIS_ADMIN_PASSWORD=$flowPassword \
--from-literal CBF_DOIS_PASSWORD=$doisPassword

helm repo add cloudbees https://charts.cloudbees.com/public/cloudbees
helm repo update
helm search repo cloudbees/cloudbees-flow --versions

helm install $release cloudbees/cloudbees-flow --version $releaseVersion \
--namespace $namespace \
--set storage.volumes.serverPlugins.storageClass=$storageClass \
--set storage.volumes.repositoryStorage.storageClass=$storageClass \
--set storage.volumes.doisStorage.storageClass=$storageClass \
--set database.externalEndpoint=$dbEndpoint \
--set database.dbPort=$dbPort \
--set database.dbType=$dbType \
--set database.dbName=$dbName \
--set database.dbUser=$dbUser \
--set database.dbPassword=$dbPass \
--set flowCredentials.existingSecret=$release-cloudbees-flow-credentials \
--set dois.credentials.existingSecret=$release-cloudbees-flow-dois \
--set serverName=$hostname \
--set ingress.host=$hostname \
--set ingress.class=$release-nginx \
--set ingress-nginx.controller.ingressClassResource.name=$release-nginx \
--set ingress-nginx.controller.ingressClassResource.enabled=true \
--set ingress-nginx.controller.ingressClassResource.default=false \
--set ingress-nginx.controller.ingressClassResource.controllerValue="k8s.io/$release-ingress-nginx" \
--set-file flowLicense.licenseData=$licenseFile \
--set-file server.customConfig.passkey\\.b64=passkey.b64 \
--set-file server.customConfig.keystore\\.b64=keystore.b64 \
--timeout 4200s▼

How to install custom ingress controllers with CloudBees CD/RO

To configure CloudBees CD/RO to use your custom or existing ingress controller:

Configure your ingress controller manifest to expose the following required TCP ports:

TCP port 8200 is required by flow-repository.
TCP ports 8443 and 61613 are required by flow-server.

(Optional) If you are using CloudBees Analytics, TCP ports 9200 and 9500 are required by flow-devopsinsight.

If you need help configuring the ingress controller manifest, refer to Exposing TCP and UDP services in the NGINX Ingress Controller documentation. Ensure you follow the example for TCP/UDP proxy support.

Update your myvalues.yaml with your custom ingress controller values:
```
ingress:
  enabled: true
  host: <your-host>
  annotations: <your-annotations>
  class: <your-custom-ingress-class>▼
```
For a reference, refer to CloudBees CD/RO Helm chart ingress example.
In your myvalues.yaml, ensure ingress-nginx.enabled is configured as false for your installation.
```
ingress-nginx:
  enabled: false▼
```
If you do not have the ingress-nginx.enabled entry in your myvalues.yaml, by default, it is set to false in the cloudbees-flow values file.
If you have not already, apply your ingress controller manifest to your Kubernetes CloudBees CD/RO cluster.

To apply the custom ingress controller configuration, update your CloudBees CD/RO installation:

helm upgrade <server-releasename> cloudbees/cloudbees-flow \
-f <myvalues.yaml> \
--namespace <server-namespace> \
--timeout 10000s▼

How to override CloudBees CD/RO Kubernetes network policies

As of v2023.04.0, you can override network polices for CloudBees CD/RO on Kubernetes by adding custom network polices and ingress controllers to override the default network policies. This is useful to deploy custom ingress controllers for internal networking for CloudBees CD/RO components or agents.

CloudBees strongly suggests testing all network policy changes in a development environment before releasing them into your production environment.

Before you start

Before overriding the default CloudBees CD/RO network policies:

You must have a NetworkPolicy manifest(s) developed that describes network behavior for all CloudBees CD/RO components.

Example NetworkPolicy manifest

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  annotations:
    meta.helm.sh/release-name: cbflow
    meta.helm.sh/release-namespace: cbflow
  labels:
    app.kubernetes.io/managed-by: Helm
  name: web-policy
  namespace: cbflow
spec:
  ingress:
  - from:
    - ipBlock:
        cidr: 0.0.0.0/0
    - podSelector:
        matchLabels:
          app: flow-bound-agent
          release: cbflow
    - podSelector:
        matchLabels:
          app: flow-server
          release: cbflow
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          app.kubernetes.io/name: ingress-nginx
    ports:
    - port: 2080
      protocol: TCP
    - port: 2443
      protocol: TCP
  podSelector:
    matchLabels:
      app: flow-web
      release: cbflow
  policyTypes:
  - Ingress▼

Your NetworkPolicy manifest(s) must be present in your cluster. To add manifest(s) to your cluster, run:
```
kubectl apply -f <your-NetworkPolicy-manifest.yaml> \
      --namespace <development-server-namespace>▼
```
If you have more than one manifest, use additional instances of -f to include them in the previous command.

If your myvalues.yaml is based on the cloudbees-flow values file v2.23.1 or earlier, you need to update the Network isolation configuration section with new configurations. To do so:

Go to Sonatype Nexus and download the latest cloudbees-flow release.
Extract the package, and open the cloudbees-flow/values.yaml.

Find the Network isolation configuration section in the values.yaml and copy it to your myvalues.yaml. Ensure the networkIsolation.networkPolicies configuration is present.

When updating your myvalues.yaml with the latest Network isolation configuration section, ensure:

You do not overwrite any custom networkIsolation.* settings.
Your values file is valid and the indention is correct.

Failing to do so may cause your upgrade to fail or other unexpected behavior.

Override CloudBees CD/RO Kubernetes network policies

After you’ve met the requirements in Before you start, in your myvalues.yaml:

Navigate to the Network isolation configuration section.

To disable the default CloudBees CD/RO network, configure networkIsolation.networkPolicies.enable: false.

After configuring networkIsolation.networkPolicies.enable: false, CloudBees CD/RO will not create its default network policies, and depend completely on your custom policies. Failure to correctly configure your custom network policies may cause unexpected behavior in your deployment.

In the additionalIngressSelector for each CloudBees CD/RO component, provide the network policy configurations for your ingress.

Example custom cbflow-web ingress configuration

  web:
    additionalIngressSelector:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          app.kubernetes.io/name: ingress-nginx▼

Once you’ve updated your myvalues.yaml, update your flow-server deployment:

helm upgrade <development-server-releasename> cloudbees/cloudbees-flow \
      -f <myvalues.yaml> \
      --namespace <development-server-namespace> \
      --timeout 10000s▼

You can now test your custom network policies in your development environment. After testing is complete, to updated your production servers, refer to Update CloudBees CD/RO production servers.

How to install custom CloudBees Analytics init container images

CloudBees CD/RO supports integrations using custom CloudBees Analytics init container images. This section guides you though how to:

Update the CloudBees Analytics Kubernetes template for custom images.

If you are using CloudBees CD/RO v2023.08.0 or earlier, this step is required. If you are using CloudBees CD/RO v2023.10.0 and later, the CloudBees Analytics Kubernetes template includes this feature, and you should start with Add custom CloudBees Analytics images to your values file.

Add custom CloudBees Analytics images to your values file.

Update the CloudBees Analytics Kubernetes template for custom images

This task is only needed if you are using CloudBees CD/RO v2023.08.0 or earlier.

To include custom CloudBees Analytics init container images with CloudBees CD/RO installations or upgrades, you must update the charts/cloudbees-flow/templates/dois-stateful-set.yaml in your CloudBees CD/RO v2023.08.0 or earlier Helm charts. To do so:

Navigate to and open charts/cloudbees-flow/templates/dois-stateful-set.yaml.
In the dois-stateful-set.yaml, navigate to spec.initContainers.image.

Update the image key-value with:

image: {{ .Values.dois.sysctlInitContainer.imageRepository }}:{{ .Values.dois.sysctlInitContainer.imageTag }}▼

Ensure the YAML is well-formed and save.

After performing these steps, the template pulls the CloudBees Analytics init container image configured in your values file based on the values of dois.sysctlInitContainer.imageRepository and dois.sysctlInitContainer.imageTag.

Add custom CloudBees Analytics images to your values file

If you are using CloudBees CD/RO v2023.08.0 or earlier, ensure you perform the steps in Update the CloudBees Analytics Kubernetes template for custom images before continuing.

In your values file, the dois.sysctlInitContainer.imageRepository and dois.sysctlInitContainer.imageTag values are used to configure the CloudBees Analytics image to include during installations and upgrades. To set these values in your values file:

For CloudBees CD/RO v2023.08.0 or earlier:

Open your values file and search for dois:. Ensure this brings you to the CloudBees Analytics configuration section of the values file.

In the CloudBees Analytics section of your values file, add the following:

  ## Enable or disable a *privileged* `sysctlInitContainer` if
  ## the sysctl `vm.max_map_count setting` is set by another method.
  sysctlInitContainer:
    enabled: true
    ## The `sysctlInitContainer.imageRepository` sysctlInitContainer image repository.
    imageRepository: busybox
    ## The `sysctlInitContainer.imageTag` sysctlInitContainer image tag .
    imageTag: stable▼

Replace the imageRepository value with your init container image repository.
Replace imageTag value with your init container image tag.
Ensure the YAML is well-formed and save.

Now that you have updated your values file with your custom CloudBees Analytics init container image, update your CloudBees CD/RO environment to apply your image. For more information, refer to Update CloudBees CD/RO production servers.

Always ensure you perform sufficient testing in your development environment before applying your changes to production environments.

For CloudBees CD/RO v2023.10.0 or later:

Open your values file and navigate to dois.sysctlInitContainer.
Replace the imageRepository value with your init container image repository.
Replace imageTag value with your init container image tag.
Ensure the YAML is well-formed and save.

Always ensure you perform sufficient testing in your development environment before applying your changes to production environments.

How to configure custom probing values

In CloudBees CD/RO v2023.12.0 and later Helm charts, liveness and readiness probe values were added for:

CloudBees CD/RO server:
- jobInit.livenessProbe
CloudBees CD/RO web server:
- web.livenessProbe
- web.readinessProbe
CloudBees CD/RO repository server:
- repository.livenessProbe
- repository.readinessProbe

To override the defaults values in the CloudBees CD/RO Helm charts, follow the instructions below to configure custom values in your myvalues.yaml.

Configure probing values for CloudBees CD/RO server jobInit

Add a custom jobInit probe values

To configure custom jobInit.livenessProbe values:

In your myvalues.yaml, navigate to values.jobInit.

Add the following fields under jobInit:

## Kubernetes Liveness Probes:
  livenessProbe:
    initialDelaySeconds: <INITIAL-DELAY-IN-SECONDS>
    periodSeconds: <PROBE-PERIOD-IN-SECONDS>
    timeoutSeconds: <TIMEOUT-LENGTH-IN-SECONDS>▼

The default values are:

## Kubernetes Liveness Probes:
  livenessProbe:
    initialDelaySeconds: 60
    periodSeconds: 60
    timeoutSeconds: 10▼

Install or upgrade your CloudBees CD/RO instance to apply these values.

Configure probing values for CloudBees CD/RO web server

Add a custom web probe values

To configure custom web.livenessProbe or web.readinessProbe values:

In your myvalues.yaml, navigate to values.web.

Add the following applicable fields under web:

## Kubernetes Liveness Probes:
  livenessProbe:
    initialDelaySeconds: <INITIAL-DELAY-IN-SECONDS>
    periodSeconds: <PROBE-PERIOD-IN-SECONDS>
    timeoutSeconds: <TIMEOUT-LENGTH-IN-SECONDS>▼

## Kubernetes Readiness Probes:
  readinessProbe:
    initialDelaySeconds: <INITIAL-DELAY-IN-SECONDS>
    periodSeconds: <PROBE-PERIOD-IN-SECONDS>
    timeoutSeconds: <TIMEOUT-LENGTH-IN-SECONDS>
    failureThreshold: <FAILURE-THRESHOLD-IN-SECONDS>▼

The default values are:

## Kubernetes Liveness Probes:
  livenessProbe:
    initialDelaySeconds: 10
    periodSeconds: 60
    timeoutSeconds: 10
    failureThreshold: 3▼

## Kubernetes Readiness Probes:
  readinessProbe:
    initialDelaySeconds: 10
    periodSeconds: 5
    timeoutSeconds: 10
    failureThreshold: 3▼

Install or upgrade your CloudBees CD/RO instance to apply these values.

Configure probing values for CloudBees CD/RO repository server

Add a custom repository probe values

To configure custom repository.livenessProbe or repository.readinessProbe values:

In your myvalues.yaml, navigate to values.repository.