Maintaining the CloudBees Analytics server on Kubernetes

Helm chart supports creating a snapshot backup of DevOps Insight Server (DOIS) Elasticsearch (ES) indices. You can access this functionality by enabling the ES backup flag during implementation. This flag creates an extra PersistentVolumeClaim (PVC) for storing snapshots of ES indices. A cron job is also added to take a snapshot backup. By default, this snapshot backup occurs every 12 hours. You can access a snapshot backup using restore, upgrade, or restore job functionality.

Limitations

The Snapshot backup mechanism has one limitation. When running more than one ES backup, all backups must have access to the same folder. Therefore, we recommend using a shared file system when running DOIS in clustered mode.

Version compatibility

A snapshot contains a copy of the on-disk data structures that comprise an index or a data stream’s backing indices. This means that snapshots can only be restored to versions of Elasticsearch that can read the indices.

The following table indicates ES snapshot compatibility with DOIS versions.

2.x

5.x

6.x

7.x

8.x

Snapshot version 1.x

Snapshot version 2.x

Snapshot version 5.x

Snapshot version 6.x

Snapshot version 7.x

Configuring a snapshot backup

To configure a snapshot backup, you must enable Helm chart support of an ES backup and restore by adding the following information to the values.yaml configuration:

dois:
  # Enable or disable creating backup volumes for dois
  backup:
    # enables creating backup volume and deploying backup cron job
    enabled: true


    # schedule cron to create snapshot of ES indices, default to every 12 hrs
    schedule_cron: "0 */12 * * *"

    # No of Days snapshots to retain
    retentionDays: 15
    # location where ES backup volume mounts on / path. Recommended to keep it as it is.
    location: "es-backups"

    # image repository for backup/restore cron jobs.
    imageRepository: cbflow-tools

    # restoreSnapshot enable this option to restore latest snapshot from
    # snapshot repository.
    restoreSnapshot: false
    # Name of Snapshot to restore if need to restore older snapshot
    restoreSnapshotName:▼

The following is a result of the values.yaml file modifications:

An extra backup PVC is created for storing indices snapshots.
A Kubernetes CronJob is created. This CronJob:
- Runs periodically creating a snapshot of indices. The run frequency is set in schedule_cron parameter.
- Purges old snapshots according to the retentionDays parameter.

Configuring snapshot backups to an external repository

Elasticsearch stores snapshots in an off-cluster storage location in a snapshot repository. Before you can take or restore snapshots, you must register a snapshot repository on the cluster. CloudBees CD/RO supports AWS S3 and Google Cloud Storage (GCS) external repositories for snapshot backups.

To back up snapshots using an external repository:

Set the dois.backup.externalRepo.enabled parameter to true.
Set the dois.backup.externalRepo.type parameter to gcs or s3.
Set the dois.backup.externalRepo.bucketName parameter.

Create a secret key using the AWS access key, AWS secret key, or the GCS service account key:

AWS_ACCESS_KEY
AWS_SECRET_KEY

GCS_SA_KEY

# Create secret for AWS S3 with permission to read/write to bucket policy with keys AWS_ACCESS_KEY and AWS_SECRET_KEY
# e.g kubectl create secret generic s3awssecret --from-literal=AWS_ACCESS_KEY="XXXXX" --from-literal=AWS_SECRET_KEY="XXXXX"
# Create secret for GCS with permission to read/write to bucket policy with service account key file with KEY GCS_SA_KEY
# e.g kubectl create secret generic gcssasecret --from-file=GCS_SA_KEY=/tmp/gke-credentials.json▼

Set the dois.backup.externalRepo.existingSecret parameter to the value of your key or set the values on the following applicable parameters:
- dois.backup.externalRepo.secret.awsAccessKey
- dois.backup.externalRepo.secret.awsSecretKey
- dois.backup.externalRepo.secret.gcsSaKey

Set the dois.backup.externalRepo.region parameter to the location of your bucket.

dois:
  # Enable or disable creating backup volumes for dois
  backup:
    # enables creating backup volume and deploying backup cron job
    enabled: true

    # schedule cron to create snapshot of ES indices, default to every 12 hrs
    schedule_cron: "0 */12 * * *"

    # Number of days to retain snapshots
    retentionDays: 15
    # location where ES backup volume mounts on / path. Recommend using default.
    location: "es-backups"

    # image repository for backup/restore cron jobs.
    imageRepository: cbflow-tools

    # restoreSnapshot enable this option to restore the latest snapshot from
    # snapshot repository.
    restoreSnapshot: false
    # Name of snapshot to restore
    restoreSnapshotName:
    # enable external repos (AWS S3 or GCS)
    externalRepo:
      enabled: true
      # type can be s3 or gcs
      type: s3
      bucketName: <Name of AWS S3 Bucket>
      # Either specify the secret where the AWS or GCS credentials stored as per below keys  or provide in values file with secret
      # Create secret for AWS S3 with permission to read/write to bucket policy with Keys AWS_ACCESS_KEY and AWS_SECRET_KEY
      # e.g. kubectl create secret generic s3awssecret --from-literal=AWS_ACCESS_KEY="XXXXX" --from-literal=AWS_SECRET_KEY="XXXXX"
      # Create secret for GCS with permission to read/write to bucket policy with service account key file with KEY GCS_SA_KEY
      # e.g. kubectl create secret generic gcssasecret --from-file=GCS_SA_KEY=/tmp/gke-credentials.json

      existingSecret: <Existing-Secret-Name as per above description>
      secret:
        # provide only if type is s3 and existingSecret is not provided
        awsAccessKey:
        # provide only if type is s3 and existingSecret is not provided
        awsSecretKey:
        # provide only if type is gcs and existingSecret is not provided
        gcsSaKey:

      existingSecret: <Secret Name>
      secret:
        awsAccessKey:
        awsSecretKey:
      region: us-east-1▼

Restoring a snapshot backup

To restore a snapshot backup, you must set the dois.backup.restoreSnapshot parameter to true during a subsequent installation or upgrade:

Considerations:

Snapshot: You cannot restore an ES backup into a cluster running an earlier version. For example, a 7.6.0 snapshot cannot be used to restore a 7.5.0 cluster.
Indices: You cannot restore indices into a cluster that is more than one major version newer than the ES version used to create the snapshot.

# restoreSnapshot enable this option to restore latest snapshot from
# snapshot repository.
restoreSnapshot: true
# Name of Snapshot to restore if need to restore older snapshot
restoreSnapshotName:▼

dois:
  # Enable or disable creating backup volumes for dois
  backup:
    # enables creating backup volume and deploying backup cron job
    enabled: true

    # schedule cron to create snapshot of ES indices, default to every 12 hrs
    schedule_cron: "0 */12 * * *"

    # location where ES backup volume mounts on / path. Recommended to keep it as it is.
    location: "es-backups"

    # image repository for backup/restore cron jobs.
    imageRepository: cbflow-tools

    # restoreSnapshot enable this option to restore latest snapshot from
    # snapshot repository.
    restoreSnapshot: true

    # Name of Snapshot to restore if need to restore older snapshot
    restoreSnapshotName:▼

Disabling a snapshot restore on the next upgrade

To disable a snapshot restore, you must set the dois.backup.restoreSnapshot parameter to false during a subsequent installation or upgrade:

# restoreSnapshot enable this option to restore latest snapshot from
# snapshot repository.
restoreSnapshot: false
# Name of Snapshot to restore if need to restore older snapshot
restoreSnapshotName:▼