工作中遇到項目組采用statefulset部署有狀態副本集，需要存儲一些中間件應用數據，應用有多個副本，靜態pv不能滿足需求，因此需要考慮動態創建持久卷。目前開發測試環境采用nas存儲數據，已經安裝了nfs服務端以及客戶端。采用動態存儲需要提前安裝nfs工具，安裝并設置nfs存儲卷之后開始以下步驟：

首先需要在項目所在k8s集群創建storageclass，即存儲類。創建storageclass首先需要創建provisioner副本。創建副本之前需要先創建RBAC。如下

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
   name: nfs-client-provisioner-runner
rules:
   - apiGroups: [""]
     resources: ["persistentvolumes"]
     verbs: ["get", "list", "watch", "create", "delete"]
   - apiGroups: [""]
     resources: ["persistentvolumeclaims"]
     verbs: ["get", "list", "watch", "update"]
   - apiGroups: ["storage.k8s.io"]
     resources: ["storageclasses"]
     verbs: ["get", "list", "watch"]
   - apiGroups: [""]
     resources: ["events"]
     verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
    name: run-nfs-client-provisioner
subjects:
    - kind: ServiceAccount
      name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
      namespace: default
roleRef:
      kind: ClusterRole
      name: nfs-client-provisioner-runner
      apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
    name: leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
    namespace: default
rules:
    - apiGroups: [""]
      resources: ["endpoints"]
      verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
    name: leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
    namespace: default
subjects:
    - kind: ServiceAccount
      name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
      namespace: default
roleRef:
      kind: Role
      name: leader-locking-nfs-client-provisioner
      apiGroup: rbac.authorization.k8s.io

接著創建provisioner，如下：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
labels:
  app: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
     matchLabels:
        app: nfs-client-provisioner
  template:
  metadata:
      labels:
         app: nfs-client-provisioner
  spec:
     serviceAccountName: nfs-client-provisioner
     containers:
     - name: nfs-client-provisioner
       image: quay.io/external_storage/nfs-client-provisioner:latest
       volumeMounts:
       - name: nfs-client-root
         mountPath: /persistentvolumes
       env:
       - name: PROVISIONER_NAME
         value: fuseim.pri/ifs
       - name: NFS_SERVER
         value: 10.10.10.60
       - name: NFS_PATH
         value: /ifs/kubernetes
     volumes:
     - name: nfs-client-root 
       nfs:
         server: 10.10.10.60
         path: /ifs/kubernetes

PROVISIONER_NAME:provisioner的名稱，創建storageclass時需要指定；
NFS_SERVER:nfs服務器的地址；
NFS_PATH:nfs服務器開放的地址；

接著創建storageclass，如下

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
  provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
  parameters:
     archiveOnDelete: "false"
  reclaimPolicy: Delete

provisioner:必須匹配deployment中的PROVISIONER_NAME；
reclaimPolicy：策略支持三種，分別是Delete，Retain，Recycle
保持（Retain）:刪除PV后后端存儲上的數據仍然存在，如需徹底刪除則需要手動刪除后端存儲volume
刪除（Delete）：刪除被PVC釋放的PV和后端存儲volume
回收（Recycle）：保留PV，但清空PV上的數據（已廢棄）

全部部署完成之后創建一個statefulset測試一下：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx-1-14-0
  namespace: poc-5
spec:
  podManagementPolicy: OrderedReady
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      project.cpaas.io/name: poc
      service.cpaas.io/name: statefulset-nginx-1-14-0
  serviceName: ''
  template:
    metadata:
    labels:
        project.cpaas.io/name: poc
        service.cpaas.io/name: statefulset-nginx-1-14-0
    spec:
      containers:
        - image: 'pocharbor.zybank.com.cn/library/nginx.1.14.0:20200426'
          imagePullPolicy: IfNotPresent
          name: nginx-1-14-0
          resources:
            limits:
              cpu: 500m
              memory: 500Mi
            requests:
              cpu: 500m
              memory: 500Mi
          volumeMounts:
            - mountPath: /tmp
              name: test
      restartPolicy: Always
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
    - metadata:
        annotations:
          volume.beta.kubernetes.io/storage-class: managed-nfs-storage
        name: test
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi

主要是volumeClaimTemplates這部分，在annotations中配置storageclass的名稱，創建成功之后可以自動分配PVC,PV。