Maintain CloudBees Analytics server data on Kubernetes

Helm chart supports creating a snapshot backup of DevOps Insight Server (DOIS) Elasticsearch (ES) indices. You can access this functionality by enabling the ES backup flag during implementation. This flag creates an extra PersistentVolumeClaim (PVC) for storing snapshots of ES indices. A cron job is also added to take a snapshot backup. By default, this snapshot backup occurs every 12 hours. You can access a snapshot backup using restore, upgrade, or restore job functionality.

Limitations

The following are known limitations to the CloudBees Analytics backup processes:

When you run more than one ES backup, all backups must have access to the same folder. Therefore, CloudBees recommends using a shared file system when running DOIS in clustered mode.

Version compatibility

A snapshot contains a copy of the on-disk data structures that comprise an index or a data stream’s backing indices. This means that snapshots can only be restored to versions of Elasticsearch that can read the indices.

The following table indicates ES snapshot compatibility with DOIS versions.

2.x

5.x

6.x

7.x

8.x

Snapshot version 1.x

Snapshot version 2.x

Snapshot version 5.x

Snapshot version 6.x

Snapshot version 7.x

Add backups to legacy versions

In v10.4, the capability to add backups for CloudBees Analytics was added to CloudBees CD/RO values files. If you are either:

Using values files based on pre-v10.4 Helm charts, or
You did not previously configure backups in your Helm charts and want to add it as part of an upgrade,

You must follow these instructions to avoid errors related to immutable fields in the PV of the CloudBees Analytics StatefulSets:

Identify the CloudBees Analytics statefulset:
```
kubectl get statefulsets -n <namespace>▼
```
Back up the existing data from your Persistent Volume (PV). Refer to the documentation for your provider for how to back up PVs.

Backing up your existing PV is critical to avoid data loss in the event issues arise during this process.
Scale down the StatefulSet to zero replicas to ensure all pods are gracefully terminated.
```
kubectl scale statefulset <statefulset-name> --replicas=0 -n <namespace>▼
```
This helps to prevent any data corruption or incomplete processes.
Delete the existing StatefulSet.

This step will not delete the underlying PVCs or PVs if they are configured to retain data.
```
kubectl delete statefulset <statefulset-name> -n <namespace>▼
```
Follow the instructions in the corresponding section for your backup plan:
- Configuring a snapshot backup
- Configuring snapshot backups to an external repository

Configuring a snapshot backup

To configure a snapshot backup, you must enable Helm chart support of an ES backup and restore by adding the following information to your values file:

If you are configuring your CloudBees CD/RO instance to perform initial ES backups as part of an upgrade, you may need to delete the devopsinsight StatefulSet before you attempt to upgrade.

This step may be necessary to avoid errors related to immutable fields in StatefulSets, such as the dois fields. When you delete and recreate the StatefulSet, it ensures the new backup configurations are applied correctly without Kubernetes errors. For more information, refer to Add backups to legacy versions.

dois:
  # Enable or disable creating backup volumes for dois
  backup:
    # enables creating backup volume and deploying backup cron job
    enabled: true


    # schedule cron to create snapshot of ES indices, default to every 12 hrs
    schedule_cron: "0 */12 * * *"

    # No of Days snapshots to retain
    retentionDays: 15
    # location where ES backup volume mounts on / path. Recommended to keep it as it is.
    location: "/es-backups"

    # image repository for backup/restore cron jobs.
    imageRepository: cbflow-tools

    # restoreSnapshot enable this option to restore latest snapshot from
    # snapshot repository.
    restoreSnapshot: false
    # Name of Snapshot to restore if need to restore older snapshot
    restoreSnapshotName:▼

The following is a result of the values.yaml file modifications:

An extra backup PVC is created for storing indices snapshots.
A Kubernetes CronJob is created. This CronJob:
- Runs periodically creating a snapshot of indices. The run frequency is set in schedule_cron parameter.
- Purges old snapshots according to the retentionDays parameter.

To apply these configurations in your environment, you must deploy them using your helm upgrade command.

After running the helm upgrade:

Ensure the new StatefulSet is created with the updated configurations. Verify the backup cron jobs and pods are created and running correctly.

Check the logs and status of the backup pods to ensure the backups are performed as expected. Confirm that the backup PV is created and data is stored correctly.

If any issues arise during the process, use your previously backed-up data to restore CloudBees Analytics.

Configuring snapshot backups to an external repository

Elasticsearch stores snapshots in an off-cluster storage location in a snapshot repository. Before you can take or restore snapshots, you must register a snapshot repository on the cluster. CloudBees CD/RO supports AWS S3 and Google Cloud Storage (GCS) external repositories for snapshot backups.

To back up snapshots using an external repository:

Set the dois.backup.externalRepo.enabled parameter to true.
Set the dois.backup.externalRepo.type parameter to gcs or s3.
Set the dois.backup.externalRepo.bucketName parameter.

Create a secret key using the AWS access key, AWS secret key, or the GCS service account key:

AWS_ACCESS_KEY
AWS_SECRET_KEY

GCS_SA_KEY

# Create secret for AWS S3 with permission to read/write to bucket policy with keys AWS_ACCESS_KEY and AWS_SECRET_KEY
# e.g kubectl create secret generic s3awssecret --from-literal=AWS_ACCESS_KEY="XXXXX" --from-literal=AWS_SECRET_KEY="XXXXX"
# Create secret for GCS with permission to read/write to bucket policy with service account key file with KEY GCS_SA_KEY
# e.g kubectl create secret generic gcssasecret --from-file=GCS_SA_KEY=/tmp/gke-credentials.json▼

Set the dois.backup.externalRepo.existingSecret parameter to the value of your key or set the values on the following applicable parameters:
- dois.backup.externalRepo.secret.awsAccessKey
- dois.backup.externalRepo.secret.awsSecretKey
- dois.backup.externalRepo.secret.gcsSaKey

Set the dois.backup.externalRepo.region parameter to the location of your bucket.

dois:
  # Enable or disable creating backup volumes for dois
  backup:
    # enables creating backup volume and deploying backup cron job
    enabled: true

    # schedule cron to create snapshot of ES indices, default to every 12 hrs
    schedule_cron: "0 */12 * * *"

    # Number of days to retain snapshots
    retentionDays: 15
    # location where ES backup volume mounts on / path. Recommend using default.
    location: "/es-backups"

    # image repository for backup/restore cron jobs.
    imageRepository: cbflow-tools

    # restoreSnapshot enable this option to restore the latest snapshot from
    # snapshot repository.
    restoreSnapshot: false
    # Name of snapshot to restore
    restoreSnapshotName:
    # enable external repos (AWS S3 or GCS)
    externalRepo:
      enabled: true
      # type can be s3 or gcs
      type: s3
      bucketName: <Name of AWS S3 Bucket>
      # Either specify the secret where the AWS or GCS credentials stored as per below keys  or provide in values file with secret
      # Create secret for AWS S3 with permission to read/write to bucket policy with Keys AWS_ACCESS_KEY and AWS_SECRET_KEY
      # e.g. kubectl create secret generic s3awssecret --from-literal=AWS_ACCESS_KEY="XXXXX" --from-literal=AWS_SECRET_KEY="XXXXX"
      # Create secret for GCS with permission to read/write to bucket policy with service account key file with KEY GCS_SA_KEY
      # e.g. kubectl create secret generic gcssasecret --from-file=GCS_SA_KEY=/tmp/gke-credentials.json

      existingSecret: <Existing-Secret-Name as per above description>
      secret:
        ## Provide *only* if type is AWS S3.
        awsAccessKey:
        awsSecretKey:
        ## Provide *only* if type is GCS.
        gcsSaKey:

      ## Region of the AWS S3 or GCS bucket. Example: us-east-1
      region:▼

Restoring a snapshot backup

To restore a snapshot backup, you must set the dois.backup.restoreSnapshot parameter to true during a subsequent installation or upgrade:

Considerations:

Snapshot: You cannot restore an ES backup into a cluster running an earlier version. For example, a 7.6.0 snapshot cannot be used to restore a 7.5.0 cluster.
Indices: You cannot restore indices into a cluster that is more than one major version newer than the ES version used to create the snapshot.

# restoreSnapshot enable this option to restore latest snapshot from
# snapshot repository.
restoreSnapshot: true
# Name of Snapshot to restore if need to restore older snapshot
restoreSnapshotName:▼

dois:
  # Enable or disable creating backup volumes for dois
  backup:
    # enables creating backup volume and deploying backup cron job
    enabled: true

    # schedule cron to create snapshot of ES indices, default to every 12 hrs
    schedule_cron: "0 */12 * * *"

    # location where ES backup volume mounts on / path. Recommended to keep it as it is.
    location: "/es-backups"

    # image repository for backup/restore cron jobs.
    imageRepository: cbflow-tools

    # restoreSnapshot enable this option to restore latest snapshot from
    # snapshot repository.
    restoreSnapshot: true

    # Name of Snapshot to restore, if needed to restore older snapshot
    restoreSnapshotName:▼

Disabling a snapshot restore on the next upgrade

To disable a snapshot restore, you must set the dois.backup.restoreSnapshot parameter to false during a subsequent installation or upgrade:

# restoreSnapshot enable this option to restore latest snapshot from
# snapshot repository.
restoreSnapshot: false
# Name of Snapshot to restore, if needed to restore older snapshot
restoreSnapshotName:▼