Backup and restore on Kubernetes

This chapter describes how to restore manually using kubectl for CloudBees CI on modern cloud platforms deployments in the cloud.

Using a rescue-pod

The approach is to create a "rescue-pod" that mounts the operations center volume via the same persistentVolumeClaim, and restore the backup from there.

Retrieve the security context in the pod to restore

$ kubectl --namespace=cje-cluster-example get pods cjoc-0 -o jsonpath='{.spec.securityContext}'
map[fsGroup:1000]▼

Scale down operations center

$ kubectl --namespace=cje-cluster-example scale statefulset/cjoc --replicas=0
statefulset.apps "cjoc" scaled▼

List the Persistent Volume Claims

$ kubectl --namespace=cje-cluster-example get pvc
NAME                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
jenkins-home-cjoc-0   Bound     pvc-6b27e963-b770-11e8-bcbf-42010a8400c1   20Gi       RWO            standard       46d
jenkins-home-mm1-0    Bound     pvc-b2b7e305-ba66-11e8-bcbf-42010a8400c1   50Gi       RWO            standard       42d
jenkins-home-mm2-0    Bound     pvc-6561b8da-c0c8-11e8-bcbf-42010a8400c1   50Gi       RWO            standard       34d▼

Run the rescue-pod with the required pvc (jenkins-home-cjoc-0 in this example) and the required security context

$ cat <<EOF | kubectl --namespace=cje-cluster-example create -f -
kind: Pod
apiVersion: v1
metadata:
  name: rescue-pod
spec:
  securityContext:
    runAsUser: 1000
    fsGroup: 1000
  volumes:
    - name: rescue-storage
      persistentVolumeClaim:
       claimName: jenkins-home-cjoc-0
  containers:
    - name: rescue-container
      image: nginx
      command: ["/bin/sh"]
      args: ["-c", "while true; do echo hello; sleep 10;done"]
      volumeMounts:
        - mountPath: "/tmp/jenkins-home"
          name: rescue-storage
EOF
pod "rescue-pod" created▼

Move the backup file to the rescue-container

kubectl cp oc-jenkins-home.backup.tar.gz rescue-pod:/tmp/▼

(Optional) Clean the previous $JENKINS_HOME.

In the case, you have a complete copy of the $JENKINS_HOME (Manual Backup) and you wish to perform a rollback after a failed update, wipe out the old content.

# file
kubectl exec --namespace=cje-cluster-example rescue-pod -it -- find /tmp/jenkins-home -type f -name '*.*' -delete
# folders
kubectl exec --namespace=cje-cluster-example rescue-pod -it -- find /tmp/jenkins-home/ -mindepth 1 -type d -name '*' -exec rm -rf {} \;▼

Uncompress the backup file inside the cjoc Persistent Volume Claim

kubectl exec --namespace=cje-cluster-example rescue-pod -it -- tar -xzf /tmp/oc-jenkins-home.backup.tar.gz -C /tmp/jenkins-home▼

Delete the rescue-pod

kubectl --namespace=cje-cluster-example delete pod rescue-pod▼

Scale up the cjoc

kubectl --namespace=cje-cluster-example scale statefulset/cjoc --replicas=1▼