Backup and restore on Kubernetes

On this page

This chapter describes how to restore manually using kubectl for CloudBees CI on modern cloud platforms deployments in the cloud.

Using a rescue-pod

The approach is to create a "rescue-pod" that mounts the Operations Center volume via the same persistentVolumeClaim, and restore the backup from there.

  1. Check the ownership id for jenkins user and group in the pod to restore. Note that the rescue-pod does not have the same configuration as the cjoc-0, so you might need to change the ownership based on the user and group id.

    $ kubectl --namespace=cje-cluster-example exec cjoc-0 -- cat /etc/passwd | grep jenkins
    jenkins:x:1000:1000:Linux User,,,:/var/jenkins_home:/bin/bash
    $ kubectl --namespace=cje-cluster-example exec cjoc-0 -- cat /etc/group | grep jenkins
    jenkins:x:1000:jenkins
  2. Scale down the cjoc

    $ kubectl --namespace=cje-cluster-example scale statefulset/cjoc --replicas=0
    statefulset.apps "cjoc" scaled
  3. List the Persistent Volume Claims

    $ kubectl --namespace=cje-cluster-example get pvc
    NAME                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    jenkins-home-cjoc-0   Bound     pvc-6b27e963-b770-11e8-bcbf-42010a8400c1   20Gi       RWO            standard       46d
    jenkins-home-mm1-0    Bound     pvc-b2b7e305-ba66-11e8-bcbf-42010a8400c1   50Gi       RWO            standard       42d
    jenkins-home-mm2-0    Bound     pvc-6561b8da-c0c8-11e8-bcbf-42010a8400c1   50Gi       RWO            standard       34d
  4. Run the rescue-pod with the required pvc (jenkins-home-cjoc-0 in this example)

    $ cat <<EOF | kubectl --namespace=cje-cluster-example create -f -
    kind: Pod
    apiVersion: v1
    metadata:
      name: rescue-pod
    spec:
      volumes:
        - name: rescue-storage
          persistentVolumeClaim:
           claimName: jenkins-home-cjoc-0
      containers:
        - name: rescue-container
          image: nginx
          volumeMounts:
            - mountPath: "/tmp/jenkins-home"
              name: rescue-storage
    EOF
    pod "rescue-pod" created
  5. Move the backup file to the rescue-container

    kubectl cp oc-jenkins-home.backup.tar.gz rescue-pod:/tmp/
  6. (Optional) Clean the previous $JENKINS_HOME.

    In the case, you have a complete copy of the $JENKINS_HOME (Manual Backup) and you wish to perform a rollback after a failed update, wipe out the old content.

    # file
    kubectl exec --namespace=cje-cluster-example rescue-pod -it -- find /tmp/jenkins-home -type f -name '*.*' -delete
    # folders
    kubectl exec --namespace=cje-cluster-example rescue-pod -it -- find /tmp/jenkins-home/ -mindepth 1 -type d -name '*' -exec rm -rf {} \;
  7. Uncompress the backup file inside the cjoc Persistent Volume Claim

    kubectl exec --namespace=cje-cluster-example rescue-pod -it -- tar -xzf /tmp/oc-jenkins-home.backup.tar.gz -C /tmp/jenkins-home
  8. Check the permissions

    kubectl exec --namespace=cje-cluster-example rescue-pod -it -- ls -laR /tmp/jenkins-home

    In the case some files and/or folders are not owned by the jenkins user and group, set the ownership recursively with:

    kubectl exec --namespace=cje-cluster-example rescue-pod -it -- chown -R 1000:1000 /tmp/jenkins-home
    1000:1000 according to the previous step 1.
  9. Delete the rescue-pod

    kubectl --namespace=cje-cluster-example delete pod rescue-pod
  10. Scale up the cjoc

    kubectl --namespace=cje-cluster-example scale statefulset/cjoc --replicas=1