Backup and restore on Kubernetes

On this page

This chapter describes how to restore manually using kubectl for CloudBees CI on modern cloud platforms deployments in the cloud.

Using a rescue-pod

The approach is to create a "rescue-pod" that mounts the Operations Center volume via the same persistentVolumeClaim, and restore the backup from there.

  1. Retrieve the security context in the pod to restore

    $ kubectl --namespace=cje-cluster-example get pods cjoc-0 -o jsonpath='{.spec.securityContext}'
    map[fsGroup:1000]
  2. Scale down the cjoc

    $ kubectl --namespace=cje-cluster-example scale statefulset/cjoc --replicas=0
    statefulset.apps "cjoc" scaled
  3. List the Persistent Volume Claims

    $ kubectl --namespace=cje-cluster-example get pvc
    NAME                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    jenkins-home-cjoc-0   Bound     pvc-6b27e963-b770-11e8-bcbf-42010a8400c1   20Gi       RWO            standard       46d
    jenkins-home-mm1-0    Bound     pvc-b2b7e305-ba66-11e8-bcbf-42010a8400c1   50Gi       RWO            standard       42d
    jenkins-home-mm2-0    Bound     pvc-6561b8da-c0c8-11e8-bcbf-42010a8400c1   50Gi       RWO            standard       34d
  4. Run the rescue-pod with the required pvc (jenkins-home-cjoc-0 in this example) and the required security context

    $ cat <<EOF | kubectl --namespace=cje-cluster-example create -f -
    kind: Pod
    apiVersion: v1
    metadata:
      name: rescue-pod
    spec:
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
      volumes:
        - name: rescue-storage
          persistentVolumeClaim:
           claimName: jenkins-home-cjoc-0
      containers:
        - name: rescue-container
          image: nginx
          command: ["/bin/sh"]
          args: ["-c", "while true; do echo hello; sleep 10;done"]
          volumeMounts:
            - mountPath: "/tmp/jenkins-home"
              name: rescue-storage
    EOF
    pod "rescue-pod" created
  5. Move the backup file to the rescue-container

    kubectl cp oc-jenkins-home.backup.tar.gz rescue-pod:/tmp/
  6. (Optional) Clean the previous $JENKINS_HOME.

    In the case, you have a complete copy of the $JENKINS_HOME (Manual Backup) and you wish to perform a rollback after a failed update, wipe out the old content.

    # file
    kubectl exec --namespace=cje-cluster-example rescue-pod -it -- find /tmp/jenkins-home -type f -name '*.*' -delete
    # folders
    kubectl exec --namespace=cje-cluster-example rescue-pod -it -- find /tmp/jenkins-home/ -mindepth 1 -type d -name '*' -exec rm -rf {} \;
  7. Uncompress the backup file inside the cjoc Persistent Volume Claim

    kubectl exec --namespace=cje-cluster-example rescue-pod -it -- tar -xzf /tmp/oc-jenkins-home.backup.tar.gz -C /tmp/jenkins-home
  8. Delete the rescue-pod

    kubectl --namespace=cje-cluster-example delete pod rescue-pod
  9. Scale up the cjoc

    kubectl --namespace=cje-cluster-example scale statefulset/cjoc --replicas=1