kubeadm HA全記錄

  • 參考方案kubeadm-ha
  • 本文cluster-info部分為原創
  • 禁止master上發布應用與參考文不同
  1. 安裝前準備

  • CentOS Linux release 7.4.1708 (Core) 8臺，其中3臺為master1,master2,master3,node 5臺為node1~node5。

  • 條件允許的話，準備VIP一個，用戶master集群。

    Host IP
    master1 172.25.16.120
    master2 172.25.16.121
    master3 172.25.16.122
    node1 172.25.16.167
    node2 172.25.16.168
    node3 172.25.16.169
    node4 172.25.16.170
    node5 172.25.16.171
    VIP 172.25.16.228

  • 所有機器上安裝好docker-ce:17.09.0-ce, kubeadm:1.7.5, kubelet:1.7.5。

    • 注意：docker建議版本是1.12，高于1.12版本，請在docker安裝后，在每臺機器上輸入iptables -P FORWARD ACCEPT

  • 貌似docker版本高于1.12都需要做以下修改,不然檢查kubelet狀態是出錯：、

    $vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
#Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
$ systemctl daemon-reload && systemctl restart kubelet

  • master1,master2,master3上安裝kubectl:1.7.5。

  • 設置每臺機器翻墻，包括yum和docker的翻墻。yum在/etc/yum.conf里配置proxy=http://SERVER:PORT,docker在/usr/lib/systemd/system/docker.service中的[service]下添加
    Environment="NO_PROXY=localhost,127.0.0.0/8,172.0.0.0/24" Environment="HTTP_PROXY=http://SERVER:PORT/" Environment="HTTPS_PROXY=http://SERVER:PORT/"

  1. etcd集群
  • 在master1上，以docker方式啟動etcd
#!/bin/bash
docker stop etcd && docker rm etcd
rm -rf /var/lib/etcd-cluster
mkdir -p /var/lib/etcd-cluster

docker run -d \
--restart always \
-v /etc/ssl/certs:/etc/ssl/certs \
-v /var/lib/etcd-cluster:/var/lib/etcd \
-p 4001:4001 \
-p 2380:2380 \
-p 2379:2379 \
--name etcd \
gcr.io/google_containers/etcd-amd64:3.0.17 \
etcd --name=etcd0 \
--advertise-client-urls=http://172.25.16.120:2379,http://172.25.16.120:4001 \
--listen-client-urls=http://0.0.0.0:2379,http://0.0.0.0:4001 \
--initial-advertise-peer-urls=http://172.25.16.120:2380 \
--listen-peer-urls=http://0.0.0.0:2380 \
--initial-cluster-token=9477af68bbee1b9ae037d6fd9e7efefd \
--initial-cluster=etcd0=http://172.25.16.120:2380,etcd1=http://172.25.16.121:2380,etcd2=http://172.25.16.122:2380 \
--initial-cluster-state=new \
--auto-tls \
--peer-auto-tls \
--data-dir=/var/lib/etcd
  • 在master2上，以docker方式啟動etcd
#!/bin/bash
docker stop etcd && docker rm etcd
rm -rf /var/lib/etcd-cluster
mkdir -p /var/lib/etcd-cluster

docker run -d \
--restart always \
-v /etc/ssl/certs:/etc/ssl/certs \
-v /var/lib/etcd-cluster:/var/lib/etcd \
-p 4001:4001 \
-p 2380:2380 \
-p 2379:2379 \
--name etcd \
gcr.io/google_containers/etcd-amd64:3.0.17 \
etcd --name=etcd1 \
--advertise-client-urls=http://172.25.16.121:2379,http://172.25.16.120:4001 \
--listen-client-urls=http://0.0.0.0:2379,http://0.0.0.0:4001 \
--initial-advertise-peer-urls=http://172.25.16.121:2380 \
--listen-peer-urls=http://0.0.0.0:2380 \
--initial-cluster-token=9477af68bbee1b9ae037d6fd9e7efefd \
--initial-cluster=etcd0=http://172.25.16.120:2380,etcd1=http://172.25.16.121:2380,etcd2=http://172.25.16.122:2380 \
--initial-cluster-state=new \
--auto-tls \
--peer-auto-tls \
--data-dir=/var/lib/etcd
  • 在master3上，以docker方式啟動etcd
#!/bin/bash
docker stop etcd && docker rm etcd
rm -rf /var/lib/etcd-cluster
mkdir -p /var/lib/etcd-cluster

docker run -d \
--restart always \
-v /etc/ssl/certs:/etc/ssl/certs \
-v /var/lib/etcd-cluster:/var/lib/etcd \
-p 4001:4001 \
-p 2380:2380 \
-p 2379:2379 \
--name etcd \
gcr.io/google_containers/etcd-amd64:3.0.17 \
etcd --name=etcd2 \
--advertise-client-urls=http://172.25.16.122:2379,http://172.25.16.122:4001 \
--listen-client-urls=http://0.0.0.0:2379,http://0.0.0.0:4001 \
--initial-advertise-peer-urls=http://172.25.16.122:2380 \
--listen-peer-urls=http://0.0.0.0:2380 \
--initial-cluster-token=9477af68bbee1b9ae037d6fd9e7efefd \
--initial-cluster=etcd0=http://172.25.16.120:2380,etcd1=http://172.25.16.121:2380,etcd2=http://172.25.16.122:2380 \
--initial-cluster-state=new \
--auto-tls \
--peer-auto-tls \
--data-dir=/var/lib/etcd
  • 在master1,mater2,master3上檢測etcd狀態
$ docker exec -ti etcd ash

$ etcdctl member list
19dcd68c1a5b8d7d: name=etcd2 peerURLs=http://172.25.16.122:2380 clientURLs=http://172.25.16.122:2379,http://172.25.16.122:4001 isLeader=true
688e88a7e1b4e844: name=etcd0 peerURLs=http://172.25.16.120:2380 clientURLs=http://172.25.16.120:2379,http://172.25.16.120:4001 isLeader=false
692a555d87ac214c: name=etcd1 peerURLs=http://172.25.16.121:2380 clientURLs=http://172.25.16.121:2379,http://172.25.16.121:4001 isLeader=false

$ etcdctl cluster-health
member 19dcd68c1a5b8d7d is healthy: got healthy result from http://172.25.16.122:2379
member 688e88a7e1b4e844 is healthy: got healthy result from http://172.25.16.120:2379
member 692a555d87ac214c is healthy: got healthy result from http://172.25.16.121:2379
cluster is healthy
  1. 在master1上通過kubeadm安裝
  • 配置文件內容 kubeadm-init-v1.7.5.yaml
 apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
kubernetesVersion: v1.7.5
networking:
  podSubnet: 10.244.0.0/16
apiServerCertSANs:
- centos-master-1
- centos-master-2
- centos-master-3
- 172.25.16.120
- 172.25.16.121
- 172.25.16.122
- 172.25.16.228
etcd:
  endpoints:
  - http://172.25.16.120:2379
  - http://172.25.16.121:2379
  - http://172.25.16.122:2379
  • 執行kubeadm init --config=kubeadm-init-v1.7.5.yaml
  • 修改 /etc/kubernetes/manifests/kube-apiserver.yaml 
       # - --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota
   - --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds
  • 重啟服務systemctl restart docker kubelet
  • 設置kubectl環境變量KUBECONFIG
  $ vi ~/.bashrc
export KUBECONFIG=/etc/kubernetes/admin.conf

$ source ~/.bashrc
  1. 安裝flannel組件
  • 建議配置文件從網上取
  • kubectl create -f flannel-rbac.yaml
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
 name: flannel
rules:
 - apiGroups:
     - ""
   resources:
     - pods
   verbs:
     - get
 - apiGroups:
     - ""
   resources:
     - nodes
   verbs:
     - list
     - watch
 - apiGroups:
     - ""
   resources:
     - nodes/status
   verbs:
     - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
 name: flannel
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: flannel
subjects:
- kind: ServiceAccount
 name: flannel
 namespace: kube-system
  • kubectl create -f flannel.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: flannel
 namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
 name: kube-flannel-cfg
 namespace: kube-system
 labels:
   tier: node
   app: flannel
data:
 cni-conf.json: |
   {
     "name": "cbr0",
     "type": "flannel",
     "delegate": {
       "isDefaultGateway": true
     }
   }
 net-conf.json: |
   {
     "Network": "10.244.0.0/16",
     "Backend": {
       "Type": "host-gw"
     }
   }
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
 name: kube-flannel-ds
 namespace: kube-system
 labels:
   tier: node
   app: flannel
spec:
 template:
   metadata:
     labels:
       tier: node
       app: flannel
   spec:
     hostNetwork: true
     nodeSelector:
       beta.kubernetes.io/arch: amd64
     tolerations:
     - key: node-role.kubernetes.io/master
       operator: Exists
       effect: NoSchedule
     serviceAccountName: flannel
     containers:
     - name: kube-flannel
       image: quay.io/coreos/flannel:v0.8.0-amd64
       command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr"]
       securityContext:
         privileged: true
       env:
       - name: POD_NAME
         valueFrom:
           fieldRef:
             fieldPath: metadata.name
       - name: POD_NAMESPACE
         valueFrom:
           fieldRef:
             fieldPath: metadata.namespace
       volumeMounts:
       - name: run
         mountPath: /run
       - name: flannel-cfg
         mountPath: /etc/kube-flannel/
     - name: install-cni
       image: quay.io/coreos/flannel:v0.8.0-amd64
       command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]
       volumeMounts:
       - name: cni
         mountPath: /etc/cni/net.d
       - name: flannel-cfg
         mountPath: /etc/kube-flannel/
     volumes:
       - name: run
         hostPath:
           path: /run
       - name: cni
         hostPath:
           path: /etc/cni/net.d
       - name: flannel-cfg
         configMap:
           name: kube-flannel-cfg
  • 執行kubectl get pods --all-namespaces -o wide,等待所有服務為running即可。
  • 至此，單點kubernetes master配置完畢。
  1. Master HA配置
  • 把master1上的/etc/kubernetes/復制到master2、master3
scp -r /etc/kubernetes/ master2:/etc/
scp -r /etc/kubernetes/ master3:/etc/

  • 在master2、master3上重啟kubelet服務，并檢查kubelet服務狀態為active (running)
    systemctl daemon-reload && systemctl restart kubelet

  • 在master2和master3上配置kubectl的環境變量KUBECONFIG。

  • 在master2、master3檢測節點狀態，發現節點已經加進來（需要時間下載鏡像，等待一會確認狀態為Ready）

  1. 修改Master配置

    • 在master2、master3上修改kube-apiserver.yaml的配置，${HOST_IP}改為本機IP
      $ vi /etc/kubernetes/manifests/kube-apiserver.yaml
 - --advertise-address=${HOST_IP}
  • 在master2和master3上的修改kubelet.conf設置，${HOST_IP}改為本機IP
$ vi /etc/kubernetes/kubelet.conf

server: https://${HOST_IP}:6443
  • 在master2和master3上修改admin.conf，${HOST_IP}修改為本機IP地址
  $ vi /etc/kubernetes/admin.conf

  server: https://${HOST_IP}:6443
  • 在master2和master3上修改controller-manager.conf，${HOST_IP}修改為本機IP地址
  $ vi /etc/kubernetes/controller-manager.conf

  server: https://${HOST_IP}:6443
  • 在master2和master3上修改scheduler.conf，${HOST_IP}修改為本機IP地址
$ vi /etc/kubernetes/scheduler.conf

  server: https://${HOST_IP}:6443
  • 在master1、master2、master3上重啟所有服務
  $ systemctl daemon-reload && systemctl restart docker kubelet
  1. 在master1,master2,master3上安裝keepalived
  • 安裝
yum install -y keepalived
systemctl enable keepalived && systemctl restart keepalived
  • 在master1、master2、master3上設置apiserver監控腳本，當apiserver檢測失敗的時候關閉keepalived服務，轉移虛擬IP地址
$ vi /etc/keepalived/check_apiserver.sh
#!/bin/bash
err=0
for k in $( seq 1 10 )
do
    check_code=$(ps -ef|grep kube-apiserver | wc -l)
    if [ "$check_code" = "1" ]; then
        err=$(expr $err + 1)
        sleep 5
        continue
    else
        err=0
        break
    fi
done
if [ "$err" != "0" ]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi


chmod a+x /etc/keepalived/check_apiserver.sh

  • 在k8s-master1、k8s-master2、k8s-master3上查看接口名字

    $ ip a | grep 192.168.60

  • 在master1 * 、master2 * 、master3上設置keepalived，參數說明如下：

  • state ${STAT * E}：為MASTER或者 * BACKUP，只能有一個MASTER

  • interface ${IN* RFACE_NAME}：為本* 需要綁定的接口名字（通過上邊的ip a命令查看）

  • mcast_src_ip ${H * OST_IP}：為本機的IP * 地址

  • priority ${PRIORITY}* ：為優先級，例如102、101、10 * 0，優先級越高越容易選擇為MASTER，優先級不能一樣
    ${VIRTUAL_IP}：為VIP地址，這里設置為172.25.16.228。

$ vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 2
    weight -5
    fall 3  
    rise 2
}
vrrp_instance VI_1 {
    state ${STATE}
    interface ${INTERFACE_NAME}
    mcast_src_ip ${HOST_IP}
    virtual_router_id 51
    priority ${PRIORITY}
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass 4be37dc3b4c90194d1600c483e10ad1d
    }
    virtual_ipaddress {
        ${VIRTUAL_IP}
    }
    track_script {
       chk_apiserver
    }
}
  • 在master1、master2、master3上重啟keepalived服務，檢測虛擬IP地址是否生效
$ systemctl restart keepalived
$ ping 172.25.16.228
  1. kube-proxy配置
  • 在master1上修改configmap/kube-proxy的server指向keepalived的虛擬IP地址
$ kubectl edit -n kube-system configmap/kube-proxy
        server: https://192.168.60.80:8443
  • 在master1上刪除所有kube-proxy的pod，讓proxy重建
  • 在master1、master2、master3上重啟docker kubelet keepalived服務
systemctl restart docker kubelet keepalived
  • 修改cluster-info中的${HOST_IP}修改為VIP的IP。
    kubectl edit configmaps cluster-info -n kube-public

server: https://${HOST_IP}:6443
  • 至此Master HA完成
  1. 加入node
  • 在master1上查看tokenkubeadm token list
  • 在node1～node5上執行kubeadm join --token ${TOKEN} 172.25.16.228:6443
  • 在maseter1上查看nodekubectl get node，狀態為Ready則為ok。
  1. 禁止master2,master3上發布應用
kubectl taint nodes master-2 node-role.kubernetes.io/master=true:NoSchedule
kubectl taint nodes master-3 node-role.kubernetes.io/master=true:NoSchedule
最后編輯于 ：
?著作權歸作者所有,轉載或內容合作請聯系作者
平臺聲明：文章內容（如有圖片或視頻亦包括在內）由作者上傳并發布，文章內容僅代表作者本人觀點，簡書系信息發布平臺，僅提供信息存儲服務。

推薦閱讀更多精彩內容