Back up CloudBees CD/RO on Kubernetes

This page outlines the processes and steps needed to back up your CloudBees CD/RO server and help ensure critical data is not lost due to unforeseen circumstances.

Depending on your specific environment, not all information here may be relevant. Additionally, if your environment is highly-customized, the steps here may not be the most optimal, and additional solutions may be required.

Lastly, there are multiple third-party solutions for creating and restoring backups, including locally and cloud-hosted tools. CloudBees offers no support for these tools, except for the specific use cases explained in the product documentation.

Back up CloudBees CD/RO server data

There are two main processes used to back up CloudBees CD/RO server data:

Backing up your database:

The main CloudBees CD/RO operational data source comes from your database. It is critical for recovery planning to regularly back up your database. However, CloudBees does not provide options for directly backing up your database, and this must be done through your database tools.

Backing up CloudBees CD/RO instance data:

You can export your CloudBees CD/RO instance in XML using ectool export. This process can export all data in CloudBees CD/RO instances or partial data depending on your needs. For more information, refer to ectool export.

Performing a full export using ectool export can be extremely memory intensive. Instead of performing full exports regularly, CloudBees suggests to priotize exporting individual critical projects regularly.

Back up CloudBees CD/RO values files

CloudBees CD/RO Kubernetes values files, for both servers and agents, are a critical part of your overall Kubernetes environment. Tracking changes and maintaining current and correct configuration data that can be used to recover and rebuild your cluster is essential.

CloudBees strongly suggests to always maintain your CloudBees CD/RO values files within version control remote from the cluster itself. If you are running version control via a Git server directly in your cluster, CloudBees strongly suggests to either do so using persistent volumes for the server or to regularly perform backups of it.

To get a copy of the values file in your current deployment, run:

releaseName="<your-current-release>"
namespace="<your-current-release-namespace>"

helm get values $releaseName --namespace $namespace > current-values.yaml▼

Configure persistent data stores

To ensure stateful data used in your project-specific CloudBees CD/RO environment is available for recovery, it should be stored in persistent volumes (PVs). This allows clusters to access and recover data from the volume in situations where they may have failed or been accidentally deleted.

When using PVs, regulary backup the stored data to ensure minimal coverage gaps if the PV becomes unavailable.

How to pre-provision volume snapshots as a PVC in StatefulSets

This how-to describes using a pre-provisioned volume snapshot as a PersistentVolumeClaim (PVC) within a PersistentVolume (PV) for a Kubernetes StatefulSet. The instructions on how to perform these actions may differ depending on your cloud provider, however the general steps are:

Create a PVC manifest.
Create a PV manifest that references your PVC and snapshot.
Apply these manifests to your cluster.
Test your cluster to ensure the PV and PVC are present with the desired values.

Even though this how-to describes specific steps for CloudBees Analytics (flow-analytics), you can modify the steps here to apply to other CloudBees CD/RO components.

Create a PVC manifest (pvc.yaml):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: PVCNAME
spec:
  storageClassName: STORAGE_CLASSS
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: STORAGE_SIZE▼

Create a PV manifest (pv.yaml) that references the PVCNAME from your PVC manifest and your snapshot:

The following example is based on using GCP’s gcePersistentDisk. Use the format required by your provider to create a reference for your snapshot.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: PVNAME
spec:
  storageClassName: STORAGE_CLASSS
  capacity:
    storage: STORAGE_SIZE
  accessModes:
    - ReadWriteOnce
  claimRef:
    namespace: NAMESPACE
    name: PVCNAME
# Use the directive from your provider to reference your snapshot
  gcePersistentDisk:
    pdName: CLOUD_DISK_NAME
    fsType: ext4▼

Update your cluster with the manifest files:

# Apply the PVC (pvc.yaml) and
# PV (pv.yaml) to your cluster:

kubectl apply -f pvc.yaml -f pv.yaml▼

Assign the following variables:

pvcName ="<PVCNAME-from-pvc.yaml>"
pvName ="<PVNAME-from-pv.yaml>"▼

Check if the PV is available in your cluster and has the desired values:
```
kubectl get pv $pvName▼
```
Check if the PVC is available in your cluster and has the desired values:
```
kubectl get pvc $pvcName▼
```

Back up configuration data

A core resource within your environment are the configuration data stored in ZooKeeper. CloudBees CD/RO stores two set of critical data in ZooKeeper:

Keystore files
Passkeys

CloudBees strongly suggests configuring a PV for ZooKeeper. In the event that ZooKeeper and its PV fail, a backup of these files is critical to restoring the CloudBees CD/RO instance.

Back up keystore files

As part of recovery planning, having a backup of the keystore file is critical to retaining access to your environment. Create a backup of the keystore file and regularly update it.

To create a local copy of the keystore file:

# Get the flow-server pod name
cdServerPod=$(kubectl get pod -l app=flow-server -o \
jsonpath='{.items[*].metadata.name}' -n $namespace); echo $cdServerPod

# Get the keystore file name
keystore=$(kubectl -n $namespace exec $cdServerPod -- ls /tmp/ | grep keystore); echo $keystore

# Copy the keystore file from flow-server pod to your local machine
kubectl -n $namespace cp $cdServerPod:/tmp/$keystore ./keystore

# Ensure the keystore file has been copied
ls -l | grep keystore▼

Back up passkeys

As part of recovery planning, having a backup of passkeys is critical to retaining access to your environment. Create a backup of the passkeys and regularly update it.

To create a local copy of the passkeys file:

# Get the flow-server pod name
cdServerPod=$(kubectl get pod -l app=flow-server -o \
jsonpath='{.items[*].metadata.name}' -n $namespace); echo $cdServerPod

# Get the passkey file name
passkey=$(kubectl -n $namespace exec $cdServerPod -- ls /tmp/ | grep passkey); echo $passkey

# Copy the passkey file from the flow-server pod to the local machine
kubectl -n $namespace cp $cdServerPod:/tmp/$passkey ./passkey

# Ensure the passkey file has been copied
ls -l | grep passkey▼