Managing snapshots of your Elasticsearch indices?

Issue

How to manage Snapshots of your Elasticsearch indices

Environment

CloudBees Jenkins Platform (CJP)
CloudBees Jenkins Enterprise (CJE)

Resolution

Make a Snapshot of indices

Sometimes before doing an operation over the cluster we need to make a snapshot of the data on it. To do that we can create a new snapshot repository and create a new snapshot. This script lists the available snapshot repositories.

#get all backup repositories
export ES_USR="YOUR_USERNAME"
export ES_PASSWD="YOUR_PASSWORD"
export DOMAIN="ES_URL"
curl -u $ES_USR:$ES_PASSWD -XGET "$DOMAIN/_snapshot/_all?pretty"

This script creates a new snapshot repository named backup, this repository will store their data into /usr/share/elasticsearch/snapshot/elasticsearch/backup folder.

export ES_USR="YOUR_USERNAME"
export ES_PASSWD="YOUR_PASSWORD"
export DOMAIN="ES_URL"
export REPO="backup"

curl -XPUT -u $ES_USR:$ES_PASSWD  "$DOMAIN/_snapshot/$REPO?pretty" -d '{ "type": "fs", "settings": { "compress": "true", "location": "/usr/share/elasticsearch/snapshot/elasticsearch/backup"}}'

This script creates a new snapshot named snapshot_1 into the repository backup

export ES_USR="YOUR_USERNAME"
export ES_PASSWD="YOUR_PASSWORD"
export DOMAIN="ES_URL"
export REPO="backup"
export SNAPSHOT="snapshot_1"

curl -u $ES_USR:$ES_PASSWD -XPUT "$DOMAIN/_snapshot/$REPO/$SNAPSHOT?pretty"

in some cases, this process could take more than one hour so you have to check the status of your snapshot with these commands

export ES_USR="YOUR_USERNAME"
export ES_PASSWD="YOUR_PASSWORD"
export DOMAIN="ES_URL"
export REPO="backup"
export SNAPSHOT="snapshot_1"

#status of the current snapshot
# when the snapshot it is finished it will return this
# {
#    "snapshots": []
# }
#
curl -u $ES_USR:$ES_PASSWD -XGET "$DOMAIN/_snapshot/$REPO/_status?pretty"
#status of snapshot_1, you have to check the `status field` until it will be SUCCESS/FAILED
curl -u $ES_USR:$ES_PASSWD -XGET "$DOMAIN/_snapshot/$REPO/$SNAPSHOT?pretty"

Errors trying to create a snapshot `snapshot is already running`

If you execute your snapshot command and see this error, it is because another snapshot is still running, so you can wait until this snapshot finished, or you can cancel it.

{
    "error":"RemoteTransportException[[Smuggler][inet[/172.17.0.2:9300]][cluster/snapshot/create]]; nested: ConcurrentSnapshotExecutionException[[backup:snapshot_1] a snapshot is already running]; ",
    "status":503
}

You have to check which repository is making a snapshot with these commands

export ES_USR="YOUR_USERNAME"
export ES_PASSWD="YOUR_PASSWORD"
export DOMAIN="ES_URL"
export REPO="backup"

curl -u $ES_USR:$ES_PASSWD -XGET "$DOMAIN/_snapshot/$REPO/_status?pretty"

once you have the name of the snapshot you can cancel it with this commands

export ES_USR="YOUR_USERNAME"
export ES_PASSWD="YOUR_PASSWORD"
export DOMAIN="ES_URL"
export REPO="cloudbees-analytics"
export SNAPSHOT_NAME="SET_THE_NAME_OF_SNAPSHOT_HERE"

## Cancel the snapshot

curl -PDELETE -u $ES_USR:$ES_PASSWD -m 30 "$DOMAIN/_snapshot/$REPO/$SNAPSHOT_NAME?pretty"
#check the status again
curl -u $ES_USR:$ES_PASSWD -XGET "$DOMAIN/_snapshot/$REPO/_status?pretty"

Restore a Snapshot of indices

When a disaster happens we have to restore data from the snapshot that we have. In order to do that, we have to get the list of snapshots available to restore with these commands

#obtain ES list of snapshots on cloudbees-analytics repository
export ES_USR="YOUR_USERNAME"
export ES_PASSWD="YOUR_PASSWORD"
export DOMAIN="ES_URL"
export REPO="cloudbees-analytics"

curl -u $ES_USR:$ES_PASSWD "$DOMAIN/_snapshot/$REPO/_all?pretty" > $REPO-snapshots.json

Now, we have to take a look at the cloudbees-analytics-snapshots.json file and check which snapshots and indices we want to restore, once we have done it, we can edit the following script by adding a new line restore "SNAPSHOT_NAME" "INDEX_NAME" for each index that we want to restore. This script will create a file for each snapshot with the results of the restore operation.

#set the username
export ES_USR="YOUR_USERNAME"
#set the password
export ES_PASSWD="YOUR_PASSWORD"
export ES_URL="URL_OF_ES"

export ES_CREDS="$ES_USR:$ES_PASSWD"
#name of the snapshot repository
export ES_REPO="cloudbees-analytics"

restore() {
    local ES_SNAPSHOT=$1
    local index=$2
    local FILE=restore-${ES_REPO}-${ES_SNAPSHOT}.json
    echo "Restoring $ES_REPO - $ES_SNAPSHOT - $index"
    echo "Close $index" >> $FILE
    curl -XPOST -u $ES_CREDS "$ES_URL/$index/_close?pretty" >> $FILE
    echo "Restore $ES_REPO - $ES_SNAPSHOT - $index" >> $FILE
    curl -XPOST -u $ES_CREDS "$ES_URL/_snapshot/$ES_REPO/$ES_SNAPSHOT/_restore?wait_for_completion=true&pretty" -d"{ \"indices\" : \"$index\", \"ignore_unavailable\": \"true\", \"include_global_state\": \"true\" }" >> $FILE
    EXIT=1
    while [ $EXIT -eq 1 ]; do
        echo " wait_for_completion $ES_REPO/$ES_SNAPSHOT"
        sleep 5s
        EXIT=$(curl -XGET -u $ES_CREDS "$ES_URL/_snapshot/$ES_REPO/$ES_SNAPSHOT/_status" | grep -c "IN_PROGRESS")
    done
}

#set the SNAPSHOT_NAME and INDEX_NAME you want to restore
restore "SNAPSHOT_NAME" "INDEX_NAME"

Delete snapshots

If we want to keep only an exact number of snapshots on a repository we could use this script to do that. It requires having installed jq a JSON parser. It will list all snapshots and keep only the last 30, and delete all the rest

#set the username
export ES_USR="YOUR_USERNAME"
#set the password
export ES_PASSWD="YOUR_PASSWORD"
export DOMAIN="URL_OF_ES"
#number of snapshot to keep
export LIMIT=30
export REPO="cloudbees-analytics"

export ES_SNAPSHOTS=$(curl -u admin:$ES_PASSWD -s -XGET "$ES_URL/_snapshot/$REPO/_all" | jq -r ".snapshots[:-${LIMIT}][].snapshot")

# Loop over the results and delete each snapshot
for SNAPSHOT in $ES_SNAPSHOTS
do
 echo "Deleting snapshot: $SNAPSHOT"
 curl -u $ES_USR:$ES_PASSWD -s -XDELETE "$DOMAIN/_snapshot/$REPO/$SNAPSHOT?pretty"
done