Changing NFS storage location

This chapter describes the process for changing the NFS mount used for JENKINS_HOME by the operations center in CloudBees CI for modern cloud platforms.

Prerequisites

Before starting it is assumed that there exist two different NFS mountable filesystems for JENKINS_HOME. One of these is the one currently in use by operations center, and the other is a new mount that is being changed to. These instructions refer to these as "old" and "new".

1. Create a provisioner for the new storage. It won’t be needed here as the new storage type has already been created, but it will be needed for any dynamically provisioned storage used later by the system.

kubectl apply -f new-provisioner.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  generation: 1
  labels:
    app: new-provisioner
  name: new-provisioner
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: new-provisioner
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: new-provisioner
    spec:
      containers:
      - env:
        - name: PROVISIONER_NAME
          value: new-provisioner
        - name: NFS_SERVER
          value: 10.2.3.4
        - name: NFS_PATH
          value: /
        image: quay.io/external_storage/new-provisioner:latest
        imagePullPolicy: Always
        name: new-provisioner
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /persistentvolumes
          name: nfs-client-root
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: new-provisioner
      serviceAccountName: new-provisioner
      terminationGracePeriodSeconds: 30
      volumes:
      - name: nfs-client-root
        nfs:
          path: /
          server: 172.20.55.134
status: {}

2. Create a new storage class in Kubernetes for the new filesystem. Apply a file like this one with the kubectl apply -f new-sc.yaml command.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: new-sc
provisioner: new-provisioner
reclaimPolicy: Delete
volumeBindingMode: Immediate

To get a copy of any current Kubernetes object’s YAML run the kubectl get --export=true -o yaml <type>/<object> command.

3. Create a new persistent volume. Apply a file like this with one with the kubectl apply -f new-pv.yaml command after setting the path and server correctly.

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: new-sc
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-cc62a5b6-e44a-4d70-8bba-a4b56a4a0656
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi
  nfs:
    path: /example-jenkins-home-cjoc-0-pvc-174ae856-b82a-11e8-a61b-12e800f18598
    server: 1.2.3.4
  persistentVolumeReclaimPolicy: Delete
  storageClassName: new-sc
status: {}

If copying a current persistent volume’s yaml, be certain to remove the claimRef section from the spec.

Use a tool like uuidgen to create the unique identifier, the uuid will need to be changed to lower case for kubernetes.

4. To prevent updates to the file systems while working on this, stop the operations center with the kubectl scale sts/cjoc --replicas=0 command.

5. Synchronize the old and new file systems.

6. Use kubectl get --export=true -o yaml pvc/<old-pvc> to get a copy of the old Persistent Volume Claim. Remove the last-applied-configuration annotation and it will look similar to:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: nfskr
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    com.cloudbees.cje.tenant: cjoc
    com.cloudbees.cje.type: cjoc
  name: jenkins-home-cjoc-0
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: new-sc
  volumeName: pvc-cc62a5b6-e44a-4d70-8bba-a4b56a4a0656
status: {}

7. Change the spec:volumeName to the new persistent volume created in step 2.

8. Delete the current persistent volume claim with the kubectl delete pvc/jenkins-home-cjoc-0 command.

9. Create a new persistent volume claim using the pvc.yaml from the previous command with the kubectl apply -f pvc.yaml command.

10. Check that the new PVC has a status of "Bound" with the kubectl get pvc/jenkins-home-cjoc-0 command.

11. Scale the operations center back to 1 replica with the kubectl scale sts/cjoc --replicas=1 command.