The CloudBees Previews documentation provides basic installation and configuration steps. However, to use CloudBees Previews in a production environment, you will likely want to configure additional tools and integrations such as TLS, DNS, and secret management, for example.
The following sections provide an example for how you might install and manage CloudBees Previews using Helmfile. Helmfile offers a declarative way for deploying multiple Helm charts. This example configuration also uses:
Google Cloud is used in this example for the required infrastructure. If you are not using Google Cloud, you can use its free offering to test this example configuration.
The example in this topic is just one way to configure CloudBees Previews for use in a production environment. You can also use Helmfile with other cloud providers or an on-premise environment. In this case, you need to adjust the configuration to your circumstances. |
Before you begin
Before you can use Helmfile, you need to create the necessary infrastructure.
This how-to uses the gcloud
CLI to create this infrastructure.
In a production environment, the infrastructure should be managed by an infrastructure build tool like Terraform.
Set up gcloud
To get started with gcloud
follow these steps:
-
(optional) Sign up for Google Cloud Platform Free Tier.
-
Download and unzip the Google Cloud SDK.
-
Change into the unzipped directory using
cd google-cloud-sdk
. -
Run
./install.sh
to add the CLI SDK to our path. -
Run
gcloud init
to initialize the SDK.
Create cluster
Once you have configured gcloud
, you can create your Google Kubernetes Engine (GKE) cluster.
The create
command in Creating the GKE cluster can take a several minutes to complete.
project_id=$(gcloud config get-value project) (1)
zone=europe-west1-b (2)
region=${zone::${#zone}-2}
gcloud container clusters create acme-cluster \ (3)
--workload-pool=$project_id.svc.id.goog --zone=$zone \
--max-nodes=3 --enable-autoscaling
gcloud container clusters get-credentials acme-cluster \ (4)
--zone=$zone --project $project_id
1 | Stores the current Google Cloud project id in the project_id variable. |
2 | Stores zone and region in variables used for creating zoned resources. The zone must lie in the chosen region. |
3 | Creates the GKE cluster. |
4 | Configures cluster access for kubectl and helmfile . |
Create service accounts
Next, create the Google service accounts and role bindings for ExternalSecrets and cert-manager. This procedure uses Workload Identity to enable these tools' Kubernetes service accounts to use the Google Cloud API.
For each Kubernetes service account that needs to make Google API calls, the Workload Identity configuration consists of three steps:
-
Creating the Google service account.
-
Giving the Google service account the required permissions by binding it to a role.
-
Linking the Google service account to the Kubernetes service account.
external_secrets_sa_name=acme-secrets-sa-$(date +%s)
gcloud iam service-accounts create $external_secrets_sa_name \ (1)
--display-name="GSA used by External Secrets"
gcloud projects add-iam-policy-binding $project_id \ (2)
--member=serviceAccount:$external_secrets_sa_name@$project_id.iam.gserviceaccount.com \
--role=roles/secretmanager.secretAccessor
gcloud iam service-accounts add-iam-policy-binding \ (3)
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:$project_id.svc.id.goog[external-secrets/external-secrets]" \
$external_secrets_sa_name@$project_id.iam.gserviceaccount.com
1 | Creates a Google Cloud service account for External Secrets to access Google Cloud Secret Manager. |
2 | Creates IAM role binding to allow the service account to access Secret Manager. |
3 | Creates IAM role binding for Workload Identity. |
cert_manager_sa_name=acme-dns01-sa-$(date +%s)
gcloud iam service-accounts create $cert_manager_sa_name \ (1)
--display-name "GSA used for DNS01 challlenges"
gcloud projects add-iam-policy-binding $project_id \ (2)
--member serviceAccount:$cert_manager_sa_name@$project_id.iam.gserviceaccount.com \
--role roles/dns.admin
gcloud iam service-accounts add-iam-policy-binding \ (3)
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:$project_id.svc.id.goog[cert-manager/cert-manager]" \
$cert_manager_sa_name@$project_id.iam.gserviceaccount.com
1 | Creates a Google Cloud service account for cert-manager to access Google Cloud DNS. |
2 | Creates IAM role binding to allow the service account to access Cloud DNS. |
3 | Creates IAM role binding for Workload Identity. |
Create external IP
You must reserve an IP address to enable external access to the cluster. The NGINX Ingress Controller will later use this IP address.
gcloud compute addresses create acme-ip --region $region
load_balancer_ip=$(gcloud --format=json compute addresses describe acme-ip --region $region | jq -r .address)
Create DNS managed zone
Next, configure the necessary DNS settings for CloudBees Previews.
Change the domain variable to a domain you own and for which you can change the nameserver configuration. |
domain=acme.example.com
gcloud services enable dns.googleapis.com (1)
gcloud dns managed-zones create acme-dns-zone --dns-name="$domain" \ (2)
--description="$domain DNS" --dnssec-state=on --visibility=public \
--project="$project_id"
gcloud dns record-sets transaction start --zone=acme-dns-zone
gcloud dns record-sets transaction add $load_balancer_ip \ (3)
--zone=acme-dns-zone \
--name=webhook.$domain \
--type=A \
--ttl=60
gcloud dns record-sets transaction add $load_balancer_ip \ (4)
--zone=acme-dns-zone \
--name=*.previews.$domain \
--type=A \
--ttl=60
gcloud dns record-sets transaction execute --zone=acme-dns-zone
gcloud dns managed-zones describe acme-dns-zone \ (5)
--format='value(nameServers)' --project="$project_id"
1 | Ensures the DNS API is enabled. |
2 | Creates the managed zone for holding the required DNS records. |
3 | Creates an A type record for the CloudBees Previews webhook component that points to the reserved IP. |
4 | Creates a wildcard A type record used for the preview environments' URLs. |
5 | Dumps the nameservers for the managed zone to the terminal. These nameservers need to be configured with the registrar of the domain you are using. |
Create managed secrets
CloudBees Previews requires secrets for your SCM personal access tokens as well as for the webhook secrets. In this how-to, we are going to store these secrets in Google’s Secret Manager and access them with the help of Kubernetes ExternalSecrets.
gcloud services enable secretmanager.googleapis.com (1)
gcloud secrets create acme-license-cert-secret --data-file=cloudbees-ci-license.cert
gcloud secrets create acme-license-key-secret --data-file=cloudbees-ci-license.key
pat='{"username": "acme-bot", "password": "top-secret"}'
echo -n "$pat" | \ (2)
gcloud secrets create acme-personal-access-token-secret \
--replication-policy="automatic" --data-file=-
webhook_secret=$(openssl rand -base64 20)
echo -n "$webhook_secret" | \ (3)
gcloud secrets create acme-webhook-secret \
--replication-policy="automatic" --data-file=-
1 | Ensures that the Secret Manager API is enabled. |
2 | Creates the personal access token for the SCM provider. Refer to Creating a Kubernetes Secret for more information. |
3 | Creates the webhook secret. Refer to Securing your webhooks. |
Export environment variables
The setup in Helmfile picks several infrastructure-related settings up from environment variables using the requiredEnv
function.
In this last step of the infrastructure preparations, you export the required environment values needed by Helmfile.
export PROJECT_ID=$project_id
export LOAD_BALANCER_IP=$load_balancer_ip
export INGRESS_HOST=$domain
export CERT_MANAGER_SA=$cert_manager_sa_name
export EXTERNAL_SECRET_SA=$external_secrets_sa_name
Helmfile
Configuration
Two files are important for the Helmfile configuration, values.yaml and Helmfile. The values.yaml is kept simple and mainly specifies the versions of the various components. The main configuration is in Helmfile.
There are many ways to split the configurations between values.yaml and Helmfile and you can even create nested configurations. |
nginx:
version: 4.0.18
certManager:
version: 1.4.0
externalSecrets:
version: 8.0.1
previews:
version: 1.0.0
namespace: previews
licenseCertSecret: acme-license-cert-secret
licenseKeySecret: acme-license-key-secret
personalAccessTokenSecret: acme-personal-access-token-secret
webhookSecret: acme-webhook-secret
repositories: (1)
- name: acme-web
cloneURL: https://github.com/acme/acme-web.git
1 | Defines a list of repositories to be enabled for CloudBees Previews. In this setup, each configured repository will refer to the same personal access token and webhook secret. |
repositories:
- name: ingress-nginx (1)
url: https://kubernetes.github.io/ingress-nginx
- name: incubator
url: https://charts.helm.sh/incubator
- name: godaddy-external-secrets
url: https://external-secrets.github.io/kubernetes-external-secrets
- name: jetstack-cert-manager
url: https://charts.jetstack.io
- name: cloudbees
url: https://charts.cloudbees.com/public/cloudbees
environments:
default:
values:
- values.yaml
releases:
- name: nginx (2)
namespace: ingress-nginx
createNamespace: true
installed: true
wait: true
chart: ingress-nginx/ingress-nginx
version: {{ .Values.nginx.version }}
values:
- controller:
service:
loadBalancerIP: {{ requiredEnv "LOAD_BALANCER_IP" }}
- name: cert-manager (3)
namespace: cert-manager
createNamespace: true
installed: true
wait: true
chart: jetstack/cert-manager
version: {{ .Values.certManager.version }}
values:
- installCRDs: true
serviceAccount:
annotations:
iam.gke.io/gcp-service-account: {{ requiredEnv "CERT_MANAGER_SA" }}@{{ requiredEnv "PROJECT_ID" }}.iam.gserviceaccount.com
- name: cert-manager-cluster-issuer (4)
namespace: cert-manager
installed: true
wait: true
needs:
- cert-manager
chart: kubernetes-incubator/raw
values:
- resources:
- apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: acme-cluster-issuer
spec:
acme:
email: john@acme.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: acme-cluster-issuer-key
solvers:
- dns01:
cloudDNS:
project: {{ requiredEnv "PROJECT_ID" }}
- name: external-secrets (5)
namespace: external-secrets
createNamespace: true
installed: true
chart: godaddy-external-secrets/kubernetes-external-secrets
version: {{ .Values.externalSecrets.version }}
values:
- env:
GOOGLE_APPLICATION_CREDENTIALS:
POLLER_INTERVAL_MILLISECONDS: 60000
serviceAccount:
name: external-secrets
annotations:
iam.gke.io/gcp-service-account: {{ requiredEnv "EXTERNAL_SECRET_SA" }}@{{ requiredEnv "PROJECT_ID" }}.iam.gserviceaccount.com
- name: previews (6)
namespace: {{ .Values.previews.namespace }}
createNamespace: true
installed: true
needs:
- nginx
- cert-manager
- external-secrets
chart: cloudbees/cloudbees-previews
version: {{ .Values.previews.version }}
values:
- global:
license:
secret: license
analytics:
enabled: true
ingress:
host: {{ requiredEnv "INGRESS_HOST" }}
class: nginx
tlsSecret: tls-previews-system-certificate
- environments:
ingress:
host: previews.{{ requiredEnv "INGRESS_HOST" }}
class: nginx
tlsSecret: tls-previews-env-certificate
- name: previews-license
chart: kubernetes-incubator/raw
namespace: {{ .Values.previews.namespace }}
installed: true
needs:
- external-secrets
values:
- resources:
- apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
name: license
spec:
backendType: gcpSecretsManager
projectId: {{ requiredEnv "PROJECT_ID" }}
data:
- key: {{ .Values.previews.licenseCertSecret }}
name: license.cert
version: latest
- key: {{ .Values.previews.licenseKeySecret }}
name: license.key
version: latest
- name: previews-personal-access-token (7)
chart: kubernetes-incubator/raw
namespace: {{ .Values.previews.namespace }}
installed: true
needs:
- external-secrets
values:
- resources:
- apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
name: previews-scm-personal-access-token
spec:
backendType: gcpSecretsManager
projectId: {{ requiredEnv "PROJECT_ID" }}
data:
- key: {{ .Values.previews.personalAccessTokenSecret }}
name: token
version: latest
property: password
- key: {{ .Values.previews.personalAccessTokenSecret }}
name: user
version: latest
property: username
- name: previews-webhook-secret (8)
namespace: {{ .Values.previews.namespace }}
installed: true
needs:
- external-secrets
chart: kubernetes-incubator/raw
values:
- resources:
- apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
name: previews-webhook-secret
spec:
backendType: gcpSecretsManager
projectId: {{ requiredEnv "PROJECT_ID" }}
data:
- key: {{ .Values.previews.webhookSecret }}
name: secret
version: latest
- name: previews-repositories (9)
namespace: {{ .Values.previews.namespace }}
installed: true
needs:
- previews
chart: incubator/raw
values:
- values.yaml
- resources:
{{ range .Values.previews.repositories }}
- apiVersion: environment.cloudbees.com/v1alpha1
kind: GitRepository
metadata:
name: {{ .name }}-repo
spec:
url: {{ .cloneURL }}
apiTokenSecretRef:
name: previews-scm-personal-access-token
webhookSecretRef:
name: previews-webhook-secret
{{ end }}
- name: previews-system-tls-certificate (10)
namespace: {{ .Values.previews.namespace }}
installed: true
chart: incubator/raw
needs:
- cert-manager
values:
- resources:
- apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: tls-previews-system-certificate
spec:
secretName: tls-previews-system-certificate
issuerRef:
name: acme-cluster-issuer
kind: ClusterIssuer
dnsNames:
- 'webhook.{{ requiredEnv "INGRESS_HOST" }}'
- 'api.{{ requiredEnv "INGRESS_HOST" }}'
- name: previews-env-tls-certificate (11)
namespace: {{ .Values.previews.namespace }}
installed: true
chart: incubator/raw
needs:
- cert-manager
values:
- resources:
- apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: tls-previews-env-certificate
spec:
secretName: tls-previews-env-certificate
issuerRef:
name: acme-cluster-issuer
kind: ClusterIssuer
dnsNames:
- '*.previews.{{ requiredEnv "INGRESS_HOST" }}'
1 | Defines the list of required Helm Chart repositories. |
2 | Installs the NGINX Ingress Controller using the created LoadBalancer IP. |
3 | Installs cert-manager for managing TLS certificates for CloudBees Previews components as well as wildcard TLS certificate for the created preview environments. |
4 | Installs a ClusterIssuer for cert-manager using the DNS01 challenge type to create certificates. |
5 | Installs External Secrets for synchronizing personal access token and webhook secret from Google Secret Manager to Kubernetes Secrets. |
6 | Installs CloudBees Previews. The installation uses independent domains for the webhook component and the preview environments. Refer to Using a dedicated domain for environments for more information. |
7 | Defines the ExternalSecret resource for the personal access token. In this how-to the same personal access token and webhook secret is used for all created GitRepository resources. In other setups there might be multiple and one needs to adjust the setup accordingly. |
8 | Defines the ExternalSecret resource for the webhook secret. |
9 | Creates a GitRepository resource for each repository defined in values.yaml. |
10 | Creates a Certificate resource for the webhook component of CloudBees Previews. |
11 | Creates a Certificate resource for the preview environments. By adding the reflector.v1.k8s.emberstack.com annotations the TLS secret will be copied to each created preview environment namespace. |
Clean up
To clean up the created cloud resources, run:
gcloud -q projects remove-iam-policy-binding $project_id \
--member=serviceAccount:$external_secrets_sa_name@$project_id.iam.gserviceaccount.com \
--role=roles/secretmanager.secretAccessor
gcloud -q iam service-accounts delete $external_secrets_sa_name@$project_id.iam.gserviceaccount.com
gcloud -q projects remove-iam-policy-binding $project_id \
--member=serviceAccount:$cert_manager_sa_name@$project_id.iam.gserviceaccount.com \
--role=roles/dns.admin
gcloud -q iam service-accounts delete $cert_manager_sa_name@$project_id.iam.gserviceaccount.com
gcloud -q secrets delete acme-license-cert-secret
gcloud -q secrets delete acme-license-key-secret
gcloud -q secrets delete acme-personal-access-token-secret
gcloud -q secrets delete acme-webhook-secret
gcloud -q compute addresses delete acme-ip --region $region
gcloud dns record-sets transaction start --zone=acme-dns-zone
gcloud dns record-sets transaction remove $load_balancer_ip \
--zone=acme-dns-zone \
--name=webhook.$domain \
--type=A \
--ttl=60
gcloud dns record-sets transaction remove $load_balancer_ip \
--zone=acme-dns-zone \
--name=*.previews.$domain \
--type=A \
--ttl=60
gcloud dns record-sets transaction execute --zone=acme-dns-zone
gcloud -q dns managed-zones delete acme-dns-zone
gcloud -q container clusters delete acme-cluster --zone=$zone