Set up a production installation on Google Cloud

8 minute read

The CloudBees Previews documentation provides basic installation and configuration steps. However, to use CloudBees Previews in a production environment, you will likely want to configure additional tools and integrations such as TLS, DNS, and secret management, for example.

The following sections provide an example for how you might install and manage CloudBees Previews using Helmfile. Helmfile offers a declarative way for deploying multiple Helm charts. This example configuration also uses:

Google Cloud is used in this example for the required infrastructure. If you are not using Google Cloud, you can use its free offering to test this example configuration.

The example in this topic is just one way to configure CloudBees Previews for use in a production environment. You can also use Helmfile with other cloud providers or an on-premise environment. In this case, you need to adjust the configuration to your circumstances.

Before you begin

Before you can use Helmfile, you need to create the necessary infrastructure. This how-to uses the gcloud CLI to create this infrastructure. In a production environment, the infrastructure should be managed by an infrastructure build tool like Terraform.

Set up gcloud

To get started with gcloud follow these steps:

  1. (optional) Sign up for Google Cloud Platform Free Tier.

  2. Download and unzip the Google Cloud SDK.

  3. Change into the unzipped directory using cd google-cloud-sdk.

  4. Run ./install.sh to add the CLI SDK to our path.

  5. Run gcloud init to initialize the SDK.

Create cluster

Once you have configured gcloud, you can create your Google Kubernetes Engine (GKE) cluster. The create command in Creating the GKE cluster can take a several minutes to complete.

Creating the GKE cluster
project_id=$(gcloud config get-value project)                          (1)
zone=europe-west1-b                                                    (2)
region=${zone::${#zone}-2}

gcloud container clusters create acme-cluster \                        (3)
  --workload-pool=$project_id.svc.id.goog --zone=$zone \
  --max-nodes=3 --enable-autoscaling
gcloud container clusters get-credentials acme-cluster \               (4)
  --zone=$zone --project $project_id
1 Stores the current Google Cloud project id in the project_id variable.
2 Stores zone and region in variables used for creating zoned resources. The zone must lie in the chosen region.
3 Creates the GKE cluster.
4 Configures cluster access for kubectl and helmfile.

Create service accounts

Next, create the Google service accounts and role bindings for ExternalSecrets and cert-manager. This procedure uses Workload Identity to enable these tools' Kubernetes service accounts to use the Google Cloud API.

For each Kubernetes service account that needs to make Google API calls, the Workload Identity configuration consists of three steps:

  1. Creating the Google service account.

  2. Giving the Google service account the required permissions by binding it to a role.

  3. Linking the Google service account to the Kubernetes service account.

Creating service accounts and role bindings for ExternalSecrets
external_secrets_sa_name=acme-secrets-sa-$(date +%s)
gcloud iam service-accounts create $external_secrets_sa_name \         (1)
  --display-name="GSA used by External Secrets"
gcloud projects add-iam-policy-binding $project_id \                   (2)
  --member=serviceAccount:$external_secrets_sa_name@$project_id.iam.gserviceaccount.com \
  --role=roles/secretmanager.secretAccessor
gcloud iam service-accounts add-iam-policy-binding \                   (3)
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:$project_id.svc.id.goog[external-secrets/external-secrets]" \
  $external_secrets_sa_name@$project_id.iam.gserviceaccount.com
1 Creates a Google Cloud service account for External Secrets to access Google Cloud Secret Manager.
2 Creates IAM role binding to allow the service account to access Secret Manager.
3 Creates IAM role binding for Workload Identity.
Creating service accounts and role bindings for cert-manager
cert_manager_sa_name=acme-dns01-sa-$(date +%s)
gcloud iam service-accounts create $cert_manager_sa_name \             (1)
  --display-name "GSA used for DNS01 challlenges"
gcloud projects add-iam-policy-binding $project_id \                   (2)
  --member serviceAccount:$cert_manager_sa_name@$project_id.iam.gserviceaccount.com \
  --role roles/dns.admin
gcloud iam service-accounts add-iam-policy-binding \                   (3)
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:$project_id.svc.id.goog[cert-manager/cert-manager]" \
  $cert_manager_sa_name@$project_id.iam.gserviceaccount.com
1 Creates a Google Cloud service account for cert-manager to access Google Cloud DNS.
2 Creates IAM role binding to allow the service account to access Cloud DNS.
3 Creates IAM role binding for Workload Identity.

Create external IP

You must reserve an IP address to enable external access to the cluster. The NGINX Ingress Controller will later use this IP address.

Creating external load balancer IP
gcloud compute addresses create acme-ip --region $region
load_balancer_ip=$(gcloud --format=json compute addresses describe acme-ip --region $region | jq -r .address)

Create DNS managed zone

Next, configure the necessary DNS settings for CloudBees Previews.

Change the domain variable to a domain you own and for which you can change the nameserver configuration.
Creating DNS managed zone
domain=acme.example.com

gcloud services enable dns.googleapis.com                              (1)
gcloud dns managed-zones create acme-dns-zone --dns-name="$domain" \   (2)
  --description="$domain DNS" --dnssec-state=on --visibility=public \
  --project="$project_id"

gcloud dns record-sets transaction start --zone=acme-dns-zone
gcloud dns record-sets transaction add $load_balancer_ip \             (3)
   --zone=acme-dns-zone \
   --name=webhook.$domain \
   --type=A \
   --ttl=60
gcloud dns record-sets transaction add $load_balancer_ip \             (4)
   --zone=acme-dns-zone \
   --name=*.previews.$domain \
   --type=A \
   --ttl=60
gcloud dns record-sets transaction execute --zone=acme-dns-zone

gcloud dns managed-zones describe acme-dns-zone \                      (5)
  --format='value(nameServers)' --project="$project_id"
1 Ensures the DNS API is enabled.
2 Creates the managed zone for holding the required DNS records.
3 Creates an A type record for the CloudBees Previews webhook component that points to the reserved IP.
4 Creates a wildcard A type record used for the preview environments' URLs.
5 Dumps the nameservers for the managed zone to the terminal. These nameservers need to be configured with the registrar of the domain you are using.

Create managed secrets

CloudBees Previews requires secrets for your SCM personal access tokens as well as for the webhook secrets. In this how-to, we are going to store these secrets in Google’s Secret Manager and access them with the help of Kubernetes ExternalSecrets.

Creating secrets
gcloud services enable secretmanager.googleapis.com          (1)

gcloud secrets create acme-license-cert-secret --data-file=cloudbees-ci-license.cert
gcloud secrets create acme-license-key-secret --data-file=cloudbees-ci-license.key
pat='{"username": "acme-bot", "password": "top-secret"}'
echo -n "$pat" | \                                           (2)
  gcloud secrets create acme-personal-access-token-secret \
  --replication-policy="automatic" --data-file=-
webhook_secret=$(openssl rand -base64 20)
echo -n "$webhook_secret" | \                                (3)
  gcloud secrets create acme-webhook-secret \
  --replication-policy="automatic" --data-file=-
1 Ensures that the Secret Manager API is enabled.
2 Creates the personal access token for the SCM provider. Refer to Creating a Kubernetes Secret for more information.
3 Creates the webhook secret. Refer to Securing your webhooks.

Export environment variables

The setup in Helmfile picks several infrastructure-related settings up from environment variables using the requiredEnv function. In this last step of the infrastructure preparations, you export the required environment values needed by Helmfile.

Exporting environment variables needed by Helmfile
export PROJECT_ID=$project_id
export LOAD_BALANCER_IP=$load_balancer_ip
export INGRESS_HOST=$domain
export CERT_MANAGER_SA=$cert_manager_sa_name
export EXTERNAL_SECRET_SA=$external_secrets_sa_name

Helmfile

Configuration

Two files are important for the Helmfile configuration, values.yaml and Helmfile. The values.yaml is kept simple and mainly specifies the versions of the various components. The main configuration is in Helmfile.

There are many ways to split the configurations between values.yaml and Helmfile and you can even create nested configurations.
values.yaml.
nginx:
  version: 4.0.18

certManager:
  version: 1.4.0

externalSecrets:
  version: 8.0.1

previews:
  version: 1.0.0
  namespace: previews
  licenseCertSecret: acme-license-cert-secret
  licenseKeySecret: acme-license-key-secret
  personalAccessTokenSecret: acme-personal-access-token-secret
  webhookSecret: acme-webhook-secret
  repositories:                                                    (1)
    - name: acme-web
      cloneURL: https://github.com/acme/acme-web.git
1 Defines a list of repositories to be enabled for CloudBees Previews. In this setup, each configured repository will refer to the same personal access token and webhook secret.
Helmfile
repositories:
- name: ingress-nginx                                                   (1)
  url: https://kubernetes.github.io/ingress-nginx
- name: incubator
  url: https://charts.helm.sh/incubator
- name: godaddy-external-secrets
  url: https://external-secrets.github.io/kubernetes-external-secrets
- name: jetstack-cert-manager
  url: https://charts.jetstack.io
- name: cloudbees
  url: https://charts.cloudbees.com/public/cloudbees

environments:
  default:
    values:
    - values.yaml

releases:
  - name: nginx                                                          (2)
    namespace: ingress-nginx
    createNamespace: true
    installed: true
    wait: true
    chart: ingress-nginx/ingress-nginx
    version: {{ .Values.nginx.version }}
    values:
      - controller:
          service:
            loadBalancerIP: {{ requiredEnv "LOAD_BALANCER_IP" }}

  - name: cert-manager                                                   (3)
    namespace: cert-manager
    createNamespace: true
    installed: true
    wait: true
    chart: jetstack/cert-manager
    version: {{ .Values.certManager.version }}
    values:
    - installCRDs: true
      serviceAccount:
        annotations:
          iam.gke.io/gcp-service-account: {{ requiredEnv "CERT_MANAGER_SA" }}@{{ requiredEnv "PROJECT_ID" }}.iam.gserviceaccount.com

  - name: cert-manager-cluster-issuer                                    (4)
    namespace: cert-manager
    installed: true
    wait: true
    needs:
      - cert-manager
    chart: kubernetes-incubator/raw
    values:
      - resources:
          - apiVersion: cert-manager.io/v1
            kind: ClusterIssuer
            metadata:
              name: acme-cluster-issuer
            spec:
              acme:
                email: john@acme.com
                server: https://acme-v02.api.letsencrypt.org/directory
                privateKeySecretRef:
                  name: acme-cluster-issuer-key
                solvers:
                  - dns01:
                      cloudDNS:
                        project: {{ requiredEnv "PROJECT_ID" }}

  - name: external-secrets                                                (5)
    namespace: external-secrets
    createNamespace: true
    installed: true
    chart: godaddy-external-secrets/kubernetes-external-secrets
    version: {{ .Values.externalSecrets.version }}
    values:
    - env:
        GOOGLE_APPLICATION_CREDENTIALS:
        POLLER_INTERVAL_MILLISECONDS: 60000
      serviceAccount:
        name: external-secrets
        annotations:
          iam.gke.io/gcp-service-account: {{ requiredEnv "EXTERNAL_SECRET_SA" }}@{{ requiredEnv "PROJECT_ID" }}.iam.gserviceaccount.com

  - name: previews                                                        (6)
    namespace: {{ .Values.previews.namespace }}
    createNamespace: true
    installed: true
    needs:
      - nginx
      - cert-manager
      - external-secrets
    chart: cloudbees/cloudbees-previews
    version: {{ .Values.previews.version }}
    values:
      - global:
          license:
            secret: license
          analytics:
            enabled: true
          ingress:
            host: {{ requiredEnv "INGRESS_HOST" }}
            class: nginx
            tlsSecret: tls-previews-system-certificate
      - environments:
          ingress:
            host: previews.{{ requiredEnv "INGRESS_HOST" }}
            class: nginx
            tlsSecret: tls-previews-env-certificate

  - name: previews-license
    chart: kubernetes-incubator/raw
    namespace: {{ .Values.previews.namespace }}
    installed: true
    needs:
      - external-secrets
    values:
    - resources:
      - apiVersion: kubernetes-client.io/v1
        kind: ExternalSecret
        metadata:
          name: license
        spec:
          backendType: gcpSecretsManager
          projectId: {{ requiredEnv "PROJECT_ID" }}
          data:
          - key: {{ .Values.previews.licenseCertSecret }}
            name: license.cert
            version: latest
          - key: {{ .Values.previews.licenseKeySecret }}
            name: license.key
            version: latest

  - name: previews-personal-access-token                                  (7)
    chart: kubernetes-incubator/raw
    namespace: {{ .Values.previews.namespace }}
    installed: true
    needs:
      - external-secrets
    values:
    - resources:
      - apiVersion: kubernetes-client.io/v1
        kind: ExternalSecret
        metadata:
          name: previews-scm-personal-access-token
        spec:
          backendType: gcpSecretsManager
          projectId: {{ requiredEnv "PROJECT_ID" }}
          data:
          - key: {{ .Values.previews.personalAccessTokenSecret }}
            name: token
            version: latest
            property: password
          - key: {{ .Values.previews.personalAccessTokenSecret }}
            name: user
            version: latest
            property: username

  - name: previews-webhook-secret                                         (8)
    namespace: {{ .Values.previews.namespace }}
    installed: true
    needs:
      - external-secrets
    chart: kubernetes-incubator/raw
    values:
      - resources:
          - apiVersion: kubernetes-client.io/v1
            kind: ExternalSecret
            metadata:
              name: previews-webhook-secret
            spec:
              backendType: gcpSecretsManager
              projectId: {{ requiredEnv "PROJECT_ID" }}
              data:
                - key: {{ .Values.previews.webhookSecret }}
                  name: secret
                  version: latest

  - name: previews-repositories                                          (9)
    namespace: {{ .Values.previews.namespace }}
    installed: true
    needs:
      - previews
    chart: incubator/raw
    values:
      - values.yaml
      - resources:
    {{ range .Values.previews.repositories }}
        - apiVersion: environment.cloudbees.com/v1alpha1
          kind: GitRepository
          metadata:
            name: {{ .name }}-repo
          spec:
            url: {{ .cloneURL }}
            apiTokenSecretRef:
              name: previews-scm-personal-access-token
            webhookSecretRef:
              name: previews-webhook-secret
    {{ end }}

  - name: previews-system-tls-certificate                                (10)
    namespace: {{ .Values.previews.namespace }}
    installed: true
    chart: incubator/raw
    needs:
      - cert-manager
    values:
    - resources:
      - apiVersion: cert-manager.io/v1
        kind: Certificate
        metadata:
          name: tls-previews-system-certificate
        spec:
          secretName: tls-previews-system-certificate
          issuerRef:
            name: acme-cluster-issuer
            kind: ClusterIssuer
          dnsNames:
          - 'webhook.{{ requiredEnv "INGRESS_HOST" }}'
          - 'api.{{ requiredEnv "INGRESS_HOST" }}'


  - name: previews-env-tls-certificate                                   (11)
    namespace: {{ .Values.previews.namespace }}
    installed: true
    chart: incubator/raw
    needs:
      - cert-manager
    values:
    - resources:
      - apiVersion: cert-manager.io/v1
        kind: Certificate
        metadata:
          name: tls-previews-env-certificate
        spec:
          secretName: tls-previews-env-certificate
          issuerRef:
            name: acme-cluster-issuer
            kind: ClusterIssuer
          dnsNames:
          - '*.previews.{{ requiredEnv "INGRESS_HOST" }}'
1 Defines the list of required Helm Chart repositories.
2 Installs the NGINX Ingress Controller using the created LoadBalancer IP.
3 Installs cert-manager for managing TLS certificates for CloudBees Previews components as well as wildcard TLS certificate for the created preview environments.
4 Installs a ClusterIssuer for cert-manager using the DNS01 challenge type to create certificates.
5 Installs External Secrets for synchronizing personal access token and webhook secret from Google Secret Manager to Kubernetes Secrets.
6 Installs CloudBees Previews. The installation uses independent domains for the webhook component and the preview environments. Refer to Using a dedicated domain for environments for more information.
7 Defines the ExternalSecret resource for the personal access token. In this how-to the same personal access token and webhook secret is used for all created GitRepository resources. In other setups there might be multiple and one needs to adjust the setup accordingly.
8 Defines the ExternalSecret resource for the webhook secret.
9 Creates a GitRepository resource for each repository defined in values.yaml.
10 Creates a Certificate resource for the webhook component of CloudBees Previews.
11 Creates a Certificate resource for the preview environments. By adding the reflector.v1.k8s.emberstack.com annotations the TLS secret will be copied to each created preview environment namespace.

Release sync

With the resources created, you can synchronize the configuration specified in the Helmfile with the cluster by running:

helmfile sync

Each time you make changes to the configuration, you rerun helmfile sync.

If helmfile sync fails, try running it again.

Release destroy

You can delete all Helm releases by running:

helmfile destroy

Clean up

To clean up the created cloud resources, run:

Cleaning up created cloud resources.
gcloud -q projects remove-iam-policy-binding $project_id \
  --member=serviceAccount:$external_secrets_sa_name@$project_id.iam.gserviceaccount.com \
  --role=roles/secretmanager.secretAccessor
gcloud -q iam service-accounts delete $external_secrets_sa_name@$project_id.iam.gserviceaccount.com

gcloud -q projects remove-iam-policy-binding $project_id \
  --member=serviceAccount:$cert_manager_sa_name@$project_id.iam.gserviceaccount.com \
  --role=roles/dns.admin
gcloud -q iam service-accounts delete $cert_manager_sa_name@$project_id.iam.gserviceaccount.com

gcloud -q secrets delete acme-license-cert-secret
gcloud -q secrets delete acme-license-key-secret
gcloud -q secrets delete acme-personal-access-token-secret
gcloud -q secrets delete acme-webhook-secret

gcloud -q compute addresses delete acme-ip --region $region

gcloud dns record-sets transaction start --zone=acme-dns-zone
gcloud dns record-sets transaction remove $load_balancer_ip \
   --zone=acme-dns-zone \
   --name=webhook.$domain \
   --type=A \
   --ttl=60
gcloud dns record-sets transaction remove $load_balancer_ip \
   --zone=acme-dns-zone \
   --name=*.previews.$domain \
   --type=A \
   --ttl=60
gcloud dns record-sets transaction execute --zone=acme-dns-zone
gcloud -q dns managed-zones delete acme-dns-zone

gcloud -q container clusters delete acme-cluster --zone=$zone
Using private image registries