Set up a production installation on Google Cloud

8 minute read

The CloudBees Previews documentation provides basic installation and configuration steps. However, to use CloudBees Previews in a production environment, you will likely want to configure additional tools and integrations such as TLS, DNS, and secret management, for example.

The following sections provide an example for how you might install and manage CloudBees Previews using Helmfile. Helmfile offers a declarative way for deploying multiple Helm charts. This example configuration also uses:

Google Cloud is used in this example for the required infrastructure. If you are not using Google Cloud, you can use its free offering to test this example configuration.

The example in this topic is just one way to configure CloudBees Previews for use in a production environment. You can also use Helmfile with other cloud providers or an on-premise environment. In this case, you need to adjust the configuration to your circumstances.

Before you begin

Before you can use Helmfile, you need to create the necessary infrastructure. This how-to uses the gcloud CLI to create this infrastructure. In a production environment, the infrastructure should be managed by an infrastructure build tool like Terraform.

Set up gcloud

To get started with gcloud follow these steps:

  1. (optional) Sign up for Google Cloud Platform Free Tier.

  2. Download and unzip the Google Cloud SDK.

  3. Change into the unzipped directory using cd google-cloud-sdk.

  4. Run ./install.sh to add the CLI SDK to our path.

  5. Run gcloud init to initialize the SDK.

Create cluster

Once you have configured gcloud, you can create your Google Kubernetes Engine (GKE) cluster. The create command in Creating the GKE cluster can take a several minutes to complete.

Creating the GKE cluster
project_id=$(gcloud config get-value project) (1) zone=europe-west1-b (2) region=${zone::${#zone}-2} gcloud container clusters create acme-cluster \ (3) --workload-pool=$project_id.svc.id.goog --zone=$zone \ --max-nodes=3 --enable-autoscaling gcloud container clusters get-credentials acme-cluster \ (4) --zone=$zone --project $project_id
1 Stores the current Google Cloud project id in the project_id variable.
2 Stores zone and region in variables used for creating zoned resources. The zone must lie in the chosen region.
3 Creates the GKE cluster.
4 Configures cluster access for kubectl and helmfile.

Create service accounts

Next, create the Google service accounts and role bindings for ExternalSecrets and cert-manager. This procedure uses Workload Identity to enable these tools' Kubernetes service accounts to use the Google Cloud API.

For each Kubernetes service account that needs to make Google API calls, the Workload Identity configuration consists of three steps:

  1. Creating the Google service account.

  2. Giving the Google service account the required permissions by binding it to a role.

  3. Linking the Google service account to the Kubernetes service account.

Creating service accounts and role bindings for ExternalSecrets
external_secrets_sa_name=acme-secrets-sa-$(date +%s) gcloud iam service-accounts create $external_secrets_sa_name \ (1) --display-name="GSA used by External Secrets" gcloud projects add-iam-policy-binding $project_id \ (2) --member=serviceAccount:$external_secrets_sa_name@$project_id.iam.gserviceaccount.com \ --role=roles/secretmanager.secretAccessor gcloud iam service-accounts add-iam-policy-binding \ (3) --role roles/iam.workloadIdentityUser \ --member "serviceAccount:$project_id.svc.id.goog[external-secrets/external-secrets]" \ $external_secrets_sa_name@$project_id.iam.gserviceaccount.com
1 Creates a Google Cloud service account for External Secrets to access Google Cloud Secret Manager.
2 Creates IAM role binding to allow the service account to access Secret Manager.
3 Creates IAM role binding for Workload Identity.
Creating service accounts and role bindings for cert-manager
cert_manager_sa_name=acme-dns01-sa-$(date +%s) gcloud iam service-accounts create $cert_manager_sa_name \ (1) --display-name "GSA used for DNS01 challlenges" gcloud projects add-iam-policy-binding $project_id \ (2) --member serviceAccount:$cert_manager_sa_name@$project_id.iam.gserviceaccount.com \ --role roles/dns.admin gcloud iam service-accounts add-iam-policy-binding \ (3) --role roles/iam.workloadIdentityUser \ --member "serviceAccount:$project_id.svc.id.goog[cert-manager/cert-manager]" \ $cert_manager_sa_name@$project_id.iam.gserviceaccount.com
1 Creates a Google Cloud service account for cert-manager to access Google Cloud DNS.
2 Creates IAM role binding to allow the service account to access Cloud DNS.
3 Creates IAM role binding for Workload Identity.

Create external IP

You must reserve an IP address to enable external access to the cluster. The NGINX Ingress Controller will later use this IP address.

Creating external load balancer IP
gcloud compute addresses create acme-ip --region $region load_balancer_ip=$(gcloud --format=json compute addresses describe acme-ip --region $region | jq -r .address)

Create DNS managed zone

Next, configure the necessary DNS settings for CloudBees Previews.

Change the domain variable to a domain you own and for which you can change the nameserver configuration.

Creating DNS managed zone
domain=acme.example.com gcloud services enable dns.googleapis.com (1) gcloud dns managed-zones create acme-dns-zone --dns-name="$domain" \ (2) --description="$domain DNS" --dnssec-state=on --visibility=public \ --project="$project_id" gcloud dns record-sets transaction start --zone=acme-dns-zone gcloud dns record-sets transaction add $load_balancer_ip \ (3) --zone=acme-dns-zone \ --name=webhook.$domain \ --type=A \ --ttl=60 gcloud dns record-sets transaction add $load_balancer_ip \ (4) --zone=acme-dns-zone \ --name=*.previews.$domain \ --type=A \ --ttl=60 gcloud dns record-sets transaction execute --zone=acme-dns-zone gcloud dns managed-zones describe acme-dns-zone \ (5) --format='value(nameServers)' --project="$project_id"
1 Ensures the DNS API is enabled.
2 Creates the managed zone for holding the required DNS records.
3 Creates an A type record for the CloudBees Previews webhook component that points to the reserved IP.
4 Creates a wildcard A type record used for the preview environments' URLs.
5 Dumps the nameservers for the managed zone to the terminal. These nameservers need to be configured with the registrar of the domain you are using.

Create managed secrets

CloudBees Previews requires secrets for your SCM personal access tokens as well as for the webhook secrets. In this how-to, we are going to store these secrets in Google’s Secret Manager and access them with the help of Kubernetes ExternalSecrets.

Creating secrets
gcloud services enable secretmanager.googleapis.com (1) gcloud secrets create acme-license-cert-secret --data-file=cloudbees-ci-license.cert gcloud secrets create acme-license-key-secret --data-file=cloudbees-ci-license.key pat='{"username": "acme-bot", "password": "top-secret"}' echo -n "$pat" | \ (2) gcloud secrets create acme-personal-access-token-secret \ --replication-policy="automatic" --data-file=- webhook_secret=$(openssl rand -base64 20) echo -n "$webhook_secret" | \ (3) gcloud secrets create acme-webhook-secret \ --replication-policy="automatic" --data-file=-
1 Ensures that the Secret Manager API is enabled.
2 Creates the personal access token for the SCM provider. Refer to Creating a Kubernetes Secret for more information.
3 Creates the webhook secret. Refer to Securing your webhooks.

Export environment variables

The setup in Helmfile picks several infrastructure-related settings up from environment variables using the requiredEnv function. In this last step of the infrastructure preparations, you export the required environment values needed by Helmfile.

Exporting environment variables needed by Helmfile
export PROJECT_ID=$project_id export LOAD_BALANCER_IP=$load_balancer_ip export INGRESS_HOST=$domain export CERT_MANAGER_SA=$cert_manager_sa_name export EXTERNAL_SECRET_SA=$external_secrets_sa_name

Helmfile

Configuration

Two files are important for the Helmfile configuration, values.yaml and Helmfile. The values.yaml is kept simple and mainly specifies the versions of the various components. The main configuration is in Helmfile.

There are many ways to split the configurations between values.yaml and Helmfile and you can even create nested configurations.

values.yaml.
nginx: version: 4.0.18 certManager: version: 1.4.0 externalSecrets: version: 8.0.1 previews: version: 1.0.0 namespace: previews licenseCertSecret: acme-license-cert-secret licenseKeySecret: acme-license-key-secret personalAccessTokenSecret: acme-personal-access-token-secret webhookSecret: acme-webhook-secret repositories: (1) - name: acme-web cloneURL: https://github.com/acme/acme-web.git
1 Defines a list of repositories to be enabled for CloudBees Previews. In this setup, each configured repository will refer to the same personal access token and webhook secret.
Helmfile
repositories: - name: ingress-nginx (1) url: https://kubernetes.github.io/ingress-nginx - name: incubator url: https://charts.helm.sh/incubator - name: godaddy-external-secrets url: https://external-secrets.github.io/kubernetes-external-secrets - name: jetstack-cert-manager url: https://charts.jetstack.io - name: cloudbees url: https://charts.cloudbees.com/public/cloudbees environments: default: values: - values.yaml releases: - name: nginx (2) namespace: ingress-nginx createNamespace: true installed: true wait: true chart: ingress-nginx/ingress-nginx version: {{ .Values.nginx.version }} values: - controller: service: loadBalancerIP: {{ requiredEnv "LOAD_BALANCER_IP" }} - name: cert-manager (3) namespace: cert-manager createNamespace: true installed: true wait: true chart: jetstack/cert-manager version: {{ .Values.certManager.version }} values: - installCRDs: true serviceAccount: annotations: iam.gke.io/gcp-service-account: {{ requiredEnv "CERT_MANAGER_SA" }}@{{ requiredEnv "PROJECT_ID" }}.iam.gserviceaccount.com - name: cert-manager-cluster-issuer (4) namespace: cert-manager installed: true wait: true needs: - cert-manager chart: kubernetes-incubator/raw values: - resources: - apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: acme-cluster-issuer spec: acme: email: john@acme.com server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: acme-cluster-issuer-key solvers: - dns01: cloudDNS: project: {{ requiredEnv "PROJECT_ID" }} - name: external-secrets (5) namespace: external-secrets createNamespace: true installed: true chart: godaddy-external-secrets/kubernetes-external-secrets version: {{ .Values.externalSecrets.version }} values: - env: GOOGLE_APPLICATION_CREDENTIALS: POLLER_INTERVAL_MILLISECONDS: 60000 serviceAccount: name: external-secrets annotations: iam.gke.io/gcp-service-account: {{ requiredEnv "EXTERNAL_SECRET_SA" }}@{{ requiredEnv "PROJECT_ID" }}.iam.gserviceaccount.com - name: previews (6) namespace: {{ .Values.previews.namespace }} createNamespace: true installed: true needs: - nginx - cert-manager - external-secrets chart: cloudbees/cloudbees-previews version: {{ .Values.previews.version }} values: - global: license: secret: license analytics: enabled: true ingress: host: {{ requiredEnv "INGRESS_HOST" }} class: nginx tlsSecret: tls-previews-system-certificate - environments: ingress: host: previews.{{ requiredEnv "INGRESS_HOST" }} class: nginx tlsSecret: tls-previews-env-certificate - name: previews-license chart: kubernetes-incubator/raw namespace: {{ .Values.previews.namespace }} installed: true needs: - external-secrets values: - resources: - apiVersion: kubernetes-client.io/v1 kind: ExternalSecret metadata: name: license spec: backendType: gcpSecretsManager projectId: {{ requiredEnv "PROJECT_ID" }} data: - key: {{ .Values.previews.licenseCertSecret }} name: license.cert version: latest - key: {{ .Values.previews.licenseKeySecret }} name: license.key version: latest - name: previews-personal-access-token (7) chart: kubernetes-incubator/raw namespace: {{ .Values.previews.namespace }} installed: true needs: - external-secrets values: - resources: - apiVersion: kubernetes-client.io/v1 kind: ExternalSecret metadata: name: previews-scm-personal-access-token spec: backendType: gcpSecretsManager projectId: {{ requiredEnv "PROJECT_ID" }} data: - key: {{ .Values.previews.personalAccessTokenSecret }} name: token version: latest property: password - key: {{ .Values.previews.personalAccessTokenSecret }} name: user version: latest property: username - name: previews-webhook-secret (8) namespace: {{ .Values.previews.namespace }} installed: true needs: - external-secrets chart: kubernetes-incubator/raw values: - resources: - apiVersion: kubernetes-client.io/v1 kind: ExternalSecret metadata: name: previews-webhook-secret spec: backendType: gcpSecretsManager projectId: {{ requiredEnv "PROJECT_ID" }} data: - key: {{ .Values.previews.webhookSecret }} name: secret version: latest - name: previews-repositories (9) namespace: {{ .Values.previews.namespace }} installed: true needs: - previews chart: incubator/raw values: - values.yaml - resources: {{ range .Values.previews.repositories }} - apiVersion: environment.cloudbees.com/v1alpha1 kind: GitRepository metadata: name: {{ .name }}-repo spec: url: {{ .cloneURL }} apiTokenSecretRef: name: previews-scm-personal-access-token webhookSecretRef: name: previews-webhook-secret {{ end }} - name: previews-system-tls-certificate (10) namespace: {{ .Values.previews.namespace }} installed: true chart: incubator/raw needs: - cert-manager values: - resources: - apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: tls-previews-system-certificate spec: secretName: tls-previews-system-certificate issuerRef: name: acme-cluster-issuer kind: ClusterIssuer dnsNames: - 'webhook.{{ requiredEnv "INGRESS_HOST" }}' - 'api.{{ requiredEnv "INGRESS_HOST" }}' - name: previews-env-tls-certificate (11) namespace: {{ .Values.previews.namespace }} installed: true chart: incubator/raw needs: - cert-manager values: - resources: - apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: tls-previews-env-certificate spec: secretName: tls-previews-env-certificate issuerRef: name: acme-cluster-issuer kind: ClusterIssuer dnsNames: - '*.previews.{{ requiredEnv "INGRESS_HOST" }}'
1 Defines the list of required Helm Chart repositories.
2 Installs the NGINX Ingress Controller using the created LoadBalancer IP.
3 Installs cert-manager for managing TLS certificates for CloudBees Previews components as well as wildcard TLS certificate for the created preview environments.
4 Installs a ClusterIssuer for cert-manager using the DNS01 challenge type to create certificates.
5 Installs External Secrets for synchronizing personal access token and webhook secret from Google Secret Manager to Kubernetes Secrets.
6 Installs CloudBees Previews. The installation uses independent domains for the webhook component and the preview environments. Refer to Using a dedicated domain for environments for more information.
7 Defines the ExternalSecret resource for the personal access token. In this how-to the same personal access token and webhook secret is used for all created GitRepository resources. In other setups there might be multiple and one needs to adjust the setup accordingly.
8 Defines the ExternalSecret resource for the webhook secret.
9 Creates a GitRepository resource for each repository defined in values.yaml.
10 Creates a Certificate resource for the webhook component of CloudBees Previews.
11 Creates a Certificate resource for the preview environments. By adding the reflector.v1.k8s.emberstack.com annotations the TLS secret will be copied to each created preview environment namespace.

Release sync

With the resources created, you can synchronize the configuration specified in the Helmfile with the cluster by running:

helmfile sync

Each time you make changes to the configuration, you rerun helmfile sync.

If helmfile sync fails, try running it again.

Release destroy

You can delete all Helm releases by running:

helmfile destroy

Clean up

To clean up the created cloud resources, run:

Cleaning up created cloud resources.
gcloud -q projects remove-iam-policy-binding $project_id \ --member=serviceAccount:$external_secrets_sa_name@$project_id.iam.gserviceaccount.com \ --role=roles/secretmanager.secretAccessor gcloud -q iam service-accounts delete $external_secrets_sa_name@$project_id.iam.gserviceaccount.com gcloud -q projects remove-iam-policy-binding $project_id \ --member=serviceAccount:$cert_manager_sa_name@$project_id.iam.gserviceaccount.com \ --role=roles/dns.admin gcloud -q iam service-accounts delete $cert_manager_sa_name@$project_id.iam.gserviceaccount.com gcloud -q secrets delete acme-license-cert-secret gcloud -q secrets delete acme-license-key-secret gcloud -q secrets delete acme-personal-access-token-secret gcloud -q secrets delete acme-webhook-secret gcloud -q compute addresses delete acme-ip --region $region gcloud dns record-sets transaction start --zone=acme-dns-zone gcloud dns record-sets transaction remove $load_balancer_ip \ --zone=acme-dns-zone \ --name=webhook.$domain \ --type=A \ --ttl=60 gcloud dns record-sets transaction remove $load_balancer_ip \ --zone=acme-dns-zone \ --name=*.previews.$domain \ --type=A \ --ttl=60 gcloud dns record-sets transaction execute --zone=acme-dns-zone gcloud -q dns managed-zones delete acme-dns-zone gcloud -q container clusters delete acme-cluster --zone=$zone