This section describes installing CloudBees Flow on Kubernetes. These instructions assume you are familiar with Kubernetes concepts and the tooling required for establishing a Kubernetes cluster. CloudBees Flow for Kubernetes is a preview feature, providing users early access for exploration.
Prerequisites
-
Kubernetes cluster and Helm: see Supported platforms for CloudBees Flow.
-
Database instance
-
For a production environment installation, see Database instance for details.
-
For a non-production environment installation, the CloudBees Flow Helm chart automatically installs a MariaDB database instance as a built-in database.
-
Non-production environment
Cluster capacity
The table below specifies the default memory and CPU requested for each CloudBees Flow component in a non-production environment.
| CloudBees Flow component/service | CPU | Memory |
|---|---|---|
Flow Server |
Per replica: 1.5 CPU, Default number of replicas: 1 |
Per replica: 6 GiB (Memory assigned to Flow JVM: 4 GiB) |
Web Server |
0.25 CPU |
256 MiB |
DevOps Insight Server |
Per replica: 0.1 CPU, Default number of replicas: 1 |
Per replica: 2 GiB (Memory assigned to Elasticsearch JVM: 1 GiB) |
Repository Server |
0.25 CPU |
512 MiB (Memory assigned to Repository JVM: 512 MiB) |
Flow Agent (Bound) |
0.25 CPU |
512 MiB (Memory assigned to Agent JVM: 256 MiB) |
Production environment
Cluster capacity
The table below specifies the default memory and CPU requested for each CloudBees Flow component in the production environment.
| CloudBees Flow component/service | CPU | Memory |
|---|---|---|
Flow Server |
Per replica: 1.5 CPU, Default number of replicas: 1 |
Per replica: 6 GiB (Memory assigned to Flow JVM: 4 GiB) |
Web Server |
0.25 CPU |
256 MiB |
DevOps Insight Server |
Per replica: 0.1 CPU, Default number of replicas: 3 |
Per replica: 2 GiB (Memory assigned to Elasticsearch JVM: 1 GiB) |
Repository Server |
0.25 CPU |
512 MiB (Memory assigned to Repository JVM: 512 MiB) |
Flow Agent (Bound) |
0.25 CPU |
512 MiB (Memory assigned to Agent JVM: 256 MiB) |
Flow Agent (Worker) |
0.25 CPU |
512 MiB (Memory assigned to Agent JVM: 64 MiB) |
ZooKeeper |
Per replica: 0.25 CPU, Default number of replicas: 3 |
Per replica: 512 MiB |
Database instance
You must configure a database for your installation of CloudBees Flow before you initiate the installation.
CloudBees Flow supports the following external databases in a production environment:
-
MySQL 5.5.12, 5.6, 5.7
-
Clean installations of the CloudBees Flow server require the MySQL JDBC driver. See Installing the MySQL JDBC Driver .
-
For upgrades, additions to my.cnf/my.ini are required. See Installing the MySQL JDBC Driver .
-
MS SQL Server 2012 (2012 R4 is recommended), 2014, 2016, and 2017
As of version 9.2, CloudBees Flow has ended support for HTTPS connection types older than TLS 1.2. If you are using MS SQL Server 2012 or 2014, you must update it to support TLS1.2. For details, see the Microsoft TLS 1.2 support for Microsoft SQL Server web page.
-
Oracle 12c, 18c, and 19c
-
MySQL 5.5.12, 5.6, 5.7 or later
For complete information about database requirements with CloudBees Flow, see Database Requirements and Database Sizing.
Database configuration
The database can be installed inside or outside the cluster. Update the database section in the cloudbees-flow-defaults.yaml file that is specified when installing the cloudbees-flow chart. See the “Database Values” section in Configuration Values .
Persistent Storage
CloudBees Flow components need varying levels of persistent storage.
-
Non-shared storage—A disk or fie system that cannot be shared between two pods or two nodes. For example, AWS EBS, GCP Persistent Disk, Azure Disk.
-
Shared storage—A disk or fie system that can be shared between two pods or two nodes. For example, AWS EFS, GCP NFS Server, Azure Filestore.
Consult the table below for storage requirements.
| CloudBees Flow component/service | Storage type | Storage amount | Mount point |
|---|---|---|---|
Shared storage |
|||
Flow Server |
|
5 GiB |
|
Repository Server |
|
20 GiB |
|
Flow Agent (bound) |
|
5GiB |
|
Non-shared storage |
|||
DevOps Insight Server |
|
Per replica: 4, 10 GiB Default number of replicas: 3 |
|
Flow Agent (worker) |
|
5GiB |
|
Zookeeper |
|
||
Flow server persistent volume provisioning
Update the storage section in the cloudbees-flow-defaults.yaml file specified when installing the cloudbees-flow chart. See Storage values for details.
Flow agent persistent volume provisioning
Update the storage section in the cloudbees-flow-defaults.yaml file specified when installing the cloudbees-flow-agent chart. See Storage values for details.
Installation and upgrade
Create your custom values file to override Helm chart values for your installation. For your convenience, preconfigured values files are available here to get you started. Use these files as is, or use them as a starting point.
| Best practice is to make a local copy of one of the preconfigured files and modify that local copy with your settings, as needed. That way, you have a record of your configuration when it comes time to update your installation. |
| Values file | Decription |
|---|---|
|
File for use with non-production installations. It configures a single node Flow server installation with a non-production license and installs MariaDB as the database. |
|
File for use with production installations. You must configure your database, storage and Flow credentials in a local copy of this file before it can be used. |
|
File listing all Helm chart values along with their default value. Use as a reference when specifying additional configuration in your local values file. |
Add public CloudBees repository as follows:
$ helm repo add cloudbees https://charts.cloudbees.com/public/cloudbees $ helm repo update
Run the following helm command to install or upgrade.
helm [install | upgrade] <chartName> --name <releaseName> \ -f <valuesFile> --namespace <nameSpace> --timeout 10000
where:
chartName |
The name of the Helm chart. Specify one of:
|
releaseName |
A user-defined name for this particular installation. If not specified, one is generated for you. Use this name in subsequent upgrades. |
valuesFile |
The local YAML-formatted values file with your specific chart override values. Specify something like |
nameSpace |
A user-defined name for the namespace under which CloudBees Flow components are installed. |
Flow installation takes several minutes. Once all the pods are running, you can track the progress of the Flow server initialization,via the `flow-server-init-job ` job log file in the Kubernetes dashboard.
Load Balancing with Ingress
This CloudBees Flow chart provides support for load balancing with the NGINIX Ingress controller, which allows simple host and URL based HTTP routing. Note that an Ingress controller typically doesn’t eliminate the need for an external load balancer—it simply adds an additional layer of routing and control behind the load balancer. With Ingress configured, all service endpoints like web, server, repository are exposed from same domain name and load-balancer endpoint.
As a best practice, configure the Ingress controller so that all Flow services can be exposed through a single load balancer. By default, Ingress is enabled in the CloudBees Flow chart. Following is a summary of the settings:
nginx-ingress.controller.ingressClass |
Default: |
nginx-ingress.controller.publishService.enabled |
Default: |
nginx-ingress.controller.scope.enabled |
Default: |
nginx-ingress.enabled |
Default: |
nginx-ingress.tcp.61613 |
Flow server Default: |
nginx-ingress.tcp.8200 |
Flow repository Default: |
nginx-ingress.tcp.8443 |
Flow web server Default: {{ .Release.Namespace }}/flow-server:8443 |
nginx-ingress.tcp.9200 |
DevOps Insight Elasticsearch database Default: |
nginx-ingress.tcp.9500 |
DevOps Insight server Default: |
Configuration Values
Helm provides several ways to set value fields. The CloudBees Flow server and agent Helm charts provide CloudBees’s default values. Helm stores these values in the chart’s cloudbees-flow-defaults.yaml file found here .
You can override these fields as follows:
-
On the helm install command line using the
--setparameter. -
In a local
my-values.yamlfile.
| CloudBees recommends using a local values file for your installation to keep all of your CloudBees Flow environment settings. |
cloudbees-flow chart configuration values
The following table lists common configurable parameters of the cloudbees-flow ` chart and their default values. See the chart’s `cloudbees-flow-defaults.yaml for the exhaustive list of options.
| Name | Description/Default |
|---|---|
Bound agent values |
|
boundAgent.ecconfigure |
String is interpreted as if passed to ecconfigure utility within the container Default: --agentInitMemoryMB=256 --agentMaxMemoryMB=256 |
boundAgent.imageName |
Default: cbflow-agent |
boundAgent.logLevel |
Default: DEBUG |
boundAgent.replicas |
Default: 1 |
boundAgent.resources.limits.cpu |
Default: 0.25 |
boundAgent.resources.limits.memory |
Default: 1024MiB |
boundAgent.resources.requests.cpu |
Default: 0.25 |
boundAgent.resources.requests.memory |
Default: 512MiB |
Database values |
|
database.clusterEndpoint |
Use this option if your database is residing in the same Kubernetes cluster as Flow. Notation is db-service.namespace If deploying into the same namespace, .namespace component can be omitted. Default: null |
database.externalEndpoint |
Database endpoint, Default: null |
database.dbName |
Default: null |
database.dbPassword |
Default: null |
database.dbPort |
Default: null |
database.dbType |
The database type with which Flow persistence works. One of:
Default: null |
database.dbUser |
Default: null |
database.existingSecret |
Use this option if you have or are planning to deploy the credential’s secret yourself. The layout has to be the same as that of Default: null |
demoMode |
Default: false |
DevOps Insight server values |
|
dois.credentials.adminPassword dois.credentials.existingSecret |
Credentials for administrative access to Elasticsearch data. It sets a password for the Default: null |
dois.enabled |
Flag that dictates whether this workload and its accompanying services are to be installed Default: true |
dois.esClusterName |
Elasticsearch cluster name Default: null |
dois.esMinMasterNodes |
Minimum number of master-eligible nodes that must be visible in order to form an Elasticsearch cluster. Default: 1 |
dois.esNumberOfShards |
The number of primary shards that an index must have. Default: null |
dois.esRam |
Elasticsearch heap size in MB. Default: 2048 |
dois.expose |
Expose DevOps Insght as externally available services ingress needs to ports 9200 and 9500 to be open to the outer world. Default: true |
dois.imageName |
Default: cbflow-dois |
dois.lsInitRam |
Default: 512 |
dois.lsMaxRam |
Default: 1024 |
dois.replicas |
Number of Elasticsearch nodes Default: 1 |
dois.resources.limits.cpu |
Default: 1 |
dois.resources.limits.memory |
Default: 3.5GiB |
dois.resources.requests.cpu |
Default: 0.1 |
dois.resources.requests.memory |
Default: 3.5GiB |
dois.serverEndpoint |
Flow server endpoint. If installing in different namespace than server server name has to be stated in form |
dois.serviceEndpoint |
The DevOps Insigt server service endpoint to be configured on the remote CloudBees Flow server. It is assumed the remote CloudBees Flow server is located in the same Kubernetes cluster. Default: flow-devopsinsight.{{ .Release.Namespace }} |
Flow credential values |
|
flowCredentials.adminPassword |
Default: |
flowCredentials.existingSecret |
Either specify the secret where the admin user password is stored under the Default: null |
Flow server license values (optional) |
|
flowLicense.existingSecret |
Default: null |
flowLicense.licenseData |
Default: null |
Images values |
|
images.pullPolicy |
Default: |
images.repository |
Default: 547883162893.dkr.ecr.us-east-1.amazonaws.com |
images.tag |
Default: 9.2.0 |
Network isolation values |
|
networkIsolation.allowFromCidr |
The CIDR allowed to make connection to all exposed flow endpoints. Default: 0.0.0.0/0 |
Nginx-ingress values |
|
nginx-ingress.controller.ingressClass |
Default: |
nginx-ingress.controller.publishService.enabled |
Default: true |
nginx-ingress.controller.scope.enabled |
Default: true |
nginx-ingress.enabled |
Default: true |
nginx-ingress.tcp.61613 |
Default: {{ .Release.Namespace }}/flow-server:61613 |
nginx-ingress.tcp.8200 |
Default: {{ .Release.Namespace }}/flow-repository:8200 |
nginx-ingress.tcp.8443 |
Default: {{ .Release.Namespace }}/flow-server:8443 |
nginx-ingress.tcp.9200 |
Default: {{ .Release.Namespace }}/flow-devopsinsight:9200 |
nginx-ingress.tcp.9500 |
Default: {{ .Release.Namespace }}/flow-devopsinsight:9500 |
Repository values |
|
repository.ecconfigure |
String to pass to `ecconfigure ` utility. Default: --repositoryInitMemoryMB=256 --repositoryMaxMemoryMB=512 |
repository.enabled |
Flag that dictates whether this workload and its accompanying services are to be installed Default: true |
repository.expose |
Expose repository as an externally available service that ingress needs to port port 8200. Default: true |
repository.imageName |
Default: cbflow-repository |
repository.logLevel |
Default: DEBUG |
repository.replicas |
Default: 1 |
repository.resources.limits.cpu |
Default: 0.25 |
repository.resources.limits.memory |
Default: 1024MiB |
repository.resources.requests.cpu |
Default: 0.25 |
repository.resources.requests.memory |
Default: 512MiB |
repository.zoneName |
The zone, assigned to the repository instance, created in the Flow server to represent this repository instance. The zone must exist in CloudBees Flow server instance. Default: null |
Flow server values |
|
server.ecconfigure |
String to pass to `ecconfigure ` utility. Default: --serverInitMemoryMB=4096 --serverMaxMemoryMB=4096 |
server.enabled |
Flag that dictates whether the web workload and its accompanying services are to be installed. Default: true |
server.expose |
As flow requires repository to be exposed as externally available services, ingress needs to ports 8443 and 61613 to be open to the outer world. Default: true |
server.imageName |
Default: cbflow-server |
server.logLevel |
Master loglevel for com.electriccloud package Default: INFO |
server.replicas |
Default: 1 |
server.resources.limits.cpu |
Default: 4 |
server.resources.limits.memory |
Default: 6GiB |
server.resources.requests.cpu |
Default: 1.5 |
server.resources.requests.memory |
Default: 6GiB |
server.zk.host |
Default: zookeeper |
server.zk.port |
Default: 2181 |
storage.volumes.doisStorage.accessMode |
Default: ReadWriteOnce |
storage.volumes.doisStorage.name |
Default: elasticsearch-data |
storage.volumes.doisStorage.storage |
Default: 10GiB |
storage.volumes.repositoryStorage.accessMode |
Default: ReadWriteOnce |
storage.volumes.repositoryStorage.name |
Default: flow-repo-artifacts |
storage.volumes.repositoryStorage.storage |
Default: 20GiB |
storage.volumes.repositoryStorage.storageClass |
Specify non-platform-default or custom storage class. Default: null |
storage.volumes.serverPlugins.accessMode |
Default: ReadWriteMany |
storage.volumes.serverPlugins.name |
Default: flow-server-shared |
storage.volumes.serverPlugins.storage |
Default: 5GiB |
storage.volumes.serverPlugins.storageClass |
Storage class for plugins directory. Currently it has to be shared across all server and web replicas. Has to be ReadWriteMany accessible. Default: null |
Flow web server values |
|
web.ecconfigure |
String to pass to `ecconfigure ` utility. |
web.enabled |
Flag that dictates whether the web workload and its accompanying services are to be installed. Default: true |
web.imageName |
Default: cbflow-web |
web.ingress.annotations. kubernetes.io/ingress.class |
Default: flow-ingress |
web.ingress.annotations.nginx.ingress. kubernetes.io/affinity |
Default: cookie |
web.ingress.annotations.nginx.ingress. kubernetes.io/proxy-body-size |
Default: 10GB |
web.ingress.annotations.nginx.ingress. kubernetes.io/secure-backends |
Default: false |
web.ingress.annotations.nginx.ingress. kubernetes.io/session-cookie-hash |
Default: sha1 |
web.ingress.annotations.nginx.ingress. kubernetes.io/session-cookie-name |
Default: route |
web.ingress.cert.crt web.ingress.cert.key |
Certificates for WEB ingress. Normally should be set when This section is just an example that ingress can be configured with certs for TLS. Default: null |
web.ingress.enabled |
Default: true |
web.ingress.host |
Default: fbapp.ecsaas.xyz |
web.replicas |
Default: 1 |
web.resources.limits.cpu |
Default: 1 |
web.resources.limits.memory |
Default: 512MiB |
web.resources.requests.cpu |
Default: 0.25 |
web.resources.requests.memory |
Default: 256MiB |
web.service.type |
Default: ClusterIP |
Zookeeper values |
|
zookeeper.enabled |
Default: true |
zookeeper.fullnameOverride |
Default: zookeeper |
zookeeper.podLabels.mode |
Default: private |
zookeeper.podLabels.ownerApp |
Default: cloudbees-flow |
zookeeper.podLabels.role |
Default: cluster-coordinator |
zookeeper.replicaCount |
Default: 3 |
zookeeper.resources.limits.cpu |
Default: 250m |
zookeeper.resources.limits.memory |
Default: 512MiB |
zookeeper.resources.requests.cpu |
Default: 250m |
zookeeper.resources.requests.memory |
Default: 512MiB |
cloudbees-flow-agent chart configuration values
The following table lists common configurable parameters of the cloudbees-flow-agent chart and their default values. See the chart’s cloudbees-flow-defaults.yaml for the exhaustive list of options.
| Name | Description/Default |
|---|---|
ecconfigure |
String to pass to Default: --agentInitMemoryMB=16 --agentMaxMemoryMB=64 |
Flow credential values Flow server credentials to use in order to register with agent as a resource on the Flow server. The |
|
flowCredentials.existingSecret |
Default: null |
flowCredentials.password |
Default: changeme |
flowCredentials.user |
Default: admin |
Image values |
|
imageName |
Default: cbflow-agent |
images.pullPolicy |
Default: IfNotPresent |
images.repository |
Default: 547883162893.dkr.ecr.us-east-1.amazonaws.com |
images.tag |
Default: 9.2.0 |
Resource values |
|
resources.limits.cpu |
Default: 1 |
resources.limits.memory |
Default: 1024Mi |
resources.requests.cpu |
Default: 0.25 |
resources.requests.memory |
Default: 512Mi |
storage.volumes.agentWorkspace.accessMode |
Default: ReadWriteOnce |
storage.volumes.agentWorkspace.storageClass |
Use with any non platform-default or custom storage class. Default: null |
storage.volumes.agentWorkspace.name |
Default: flow-agent-workspace |
storage.volumes.agentWorkspace.storage |
Default: 5GiB |
Miscellaneous agent values |
|
logLevel |
Master loglevel for com.electriccloud package Default: DEBUG |
replicas |
Default: 1 |
resourceName |
Flow resource name that agents of this deployment are assigned to. See Agent resource name templates for more options,. Default: hostname |
resourcePools |
List of resource pools with which agents try to register upon coming up and registering with server. Default: null |
resourceType |
Type of resource to create on the remote CloudBees Flow server. This argument is relevant only when the CloudBees Flow server is using a mixed-mode license (concurrent resources and registered hosts). Valid options for this value are Default: null |
serverEndpoint |
Default: null |
trustedAgent |
Set to `true ` to configure the agent as a trusted agent, restricting the agent to one CloudBees Flow server deployment. Default: false |
workspaceName |
The default workspace (must exist in CloudBees Flow server instance) for the resources created in the Flow server for the agent(s). Default: null |
zoneName |
The zone (must exist in CloudBees Flow server instance) for the resources created in the Flow server for the agent(s). Default: null |
Agent resource name templates
Generally, you set the name for the agent resource with the value agent.resourceName, but this only works when the agent deployment has only one replica. If there are multiple replicas, then each of them tries to register itself using the same resource name. As a result, only one replica from the entire deployment will be registered as a resource.
The resource name defaults to hostname, but other template values are available to fully qualify the resource:
-
Ordinary helm template values. For example:
{{ .Release.Name }},{{ .Release.Namespace }}, and so on. -
In addition, two special template values are allowed :
-
{{ hostname }}–the actual resource hostname -
{{ ordinalIndex }}–the serial replica index in the StatefulSet
-
For example:
resourceName: "myResource - {{ .Release.Name }} - {{ ordinalIndex }}"
Troubleshooting
How to deploy monitoring support
Use Grafana to visualize the Elasticsearch metrics. Issue the following commands to deploy it at you r site.
helm install --name prometheus stable/prometheus helm install --name grafana stable/grafana # Note that username is admin and password is the result of following command kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | \ base64 --decode ; echo RandomPassword
How to deploy logging support
In general, each object in a Kubernetes cluster produces its own logs. And usually the user has their own mechanism in place to manage logs from different services in the cluster. Logs from CloudBees Flow services and pods can be captured with standard log shipper tools. A sample configuration file for the FileBeat log shipper is provided here .
How to increase memory limits for Flow components
During periods of high work load, a server component could run out of memory if it asks for more memory than is allocated to the JVM. In order to increase the memory for a component, we have to allocate more memory to the component’s container. Then, depending on the component, the memory allocation for the component, itself, running in the container needs to be increased accordingly. See Cluster capacity for default container memory settings.
The following configurations can be used to change the memory allocation for each container and component.
| Component | Container memory limit | Component memory setting | Example |
|---|---|---|---|
Flow server |
|
server.ecconfigure |
|
Flow web server |
|
N/A |
|
Repository server |
|
repository.ecconfigure |
ecconfigure: " --repositoryInitMemoryMB=256 --repositoryMaxMemoryMB=512" |
DevOps Insight server |
|
|
|
Bound agent |
|
|
|
Inject new memory limits using helm. Update your local values file (here it is called my-values.yaml ) with the new values and issue the helm `upgrade ` command.
helm upgrade <chartName> --name <releaseName> \ -f <valuesFile> --namespace <nameSpace> --timeout 10000
How to resolve volume note affinity conflicts
Sometimes pods can hang in the `pending ` stage with the following error:
x/y nodes are available: y node(s) had volume node affinity conflict.
This can happen when the availability zone for the persistent volume claim is different from the availability zone of the node on which the pod gets scheduled.
A cluster administrator can address this issue by specifying the WaitForFirstConsumer mode, which delays the binding and provisioning of a PersistentVolume ` until a pod using the `PersistentVolumeClaim is created. PersistentVolume s are selected or provisioned conforming to the topology that is specified by the pod’s scheduling constraints.
For more information see this article: https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode
Data backups and disaster recovery
Kubernetes backup tools such as Heptio’s Velero (previously known as Ark) are used to backup and restore Kubernetes clusters. CloudBees Flow services and pods deployed in the cluster can be backed using those standard backup tools. Application data maintained in persistent volumes needs to be backed up at the same time. Tools such as Velero support persistent volume backups.
Here is the list of volumes associated with the Flow pods:
| Container | Volumes |
|---|---|
flow-server |
efs-deployment, logback-config,init-scripts |
flow-agent |
flow-agent-workspace, logback-config |
flow-devopsinsight |
elasticsearch-data |
flow-repository |
efs-repository, logback-config |
flow-web |
efs-deployment |