建議使用阿里云自身的容器服務吧，畢竟自建k8s集群太折騰
1、資源架構
前期使用 3 master + 3 worker配置，后期再擴展worker節點
| 資源、主機名 | IP | 配置 | 服務 |
|---|---|---|---|
| SLB1-ALB | 172.18.20.55 | 內網基礎版 | master VIP |
| SLB2 | 外網IP，應用服務 | ingress | |
| k8s-master1 | 172.18.20.44 | ecs 4VCPU,16G + 80G | ETCD,API server, Controller Manager, Scheduler |
| k8s-master2 | 172.18.20.45 | ecs 4VCPU,16G + 80G | ETCD,API server, Controller Manager, Scheduler |
| k8s-master3 | 172.18.20.46 | ecs 4VCPU,16G + 80G | ETCD,API server, Controller Manager, Scheduler |
| k8s-worker1 | 172.18.20.47 | ecs 8vCPU,32G + 80G | kubelet, kube-proxy, docker, fluentd |
| k8s-worker2 | 172.18.20.48 | ecs 8vCPU,32G + 80G | kubelet, kube-proxy, docker, fluentd |
| k8s-worker3 | 172.18.20.49 | ecs 8vCPU,32G + 80G | kubelet, kube-proxy, docker, fluentd |
架構圖
堆疊（Stacked） etcd 拓撲
堆疊的 etcd 拓撲
外部 etcd 拓撲
外部 etcd 拓撲
2、基礎環境
2.1、軟件版本
由于kubernetes 1.20 版本開始已棄用docker，推薦使用containerd
| 軟件 | 版本 |
|---|---|
| os | CentOS 7.9 |
| containerd | 1.4.3 |
| Kubernetes | 1.21 |
| etcd | 3.4 |
2.2、系統配置
沒有特別說明的，所有主機需執行
- 修改hostname
hostname k8s-master1
- 關閉防火墻
# systemctl stop firewalld
# systemctl disable firewalld
- 關閉selinux
# sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久
# setenforce 0 # 臨時
- 關閉 swap
# swapoff -a # 臨時
# sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久
- 添加 hosts
# cat >> /etc/hosts << EOF
172.18.20.44 k8s-master1
172.18.20.45 k8s-master2
172.18.20.46 k8s-master3
...
EOF
- 配置節點主機免密訪問(master)
# ssh-keygen
# ssh-copy-id root@k8s-*
- 將橋接的IPv4流量傳遞到iptables的鏈
# cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
# sysctl --system
-
調整系統內核參數
# cat > /etc/sysctl.d/kubernetes.conf <<EOF net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv6.conf.all.disable_ipv6=1 net.ipv4.ip_forward=1 net.ipv4.tcp_tw_recycle=0 vm.swappiness=0 fs.file-max=2000000 fs.nr_open=2000000 fs.inotify.max_user_instances=512 fs.inotify.max_user_watches=1280000 net.netfilter.nf_conntrack_max=524288 EOF # modprobe br_netfilter && sysctl -p /etc/sysctl.d/kubernetes.conf -
錯誤提示
/proc/sys/net/bridge/bridge-nf-call-iptables: 沒有那個文件或目錄記得運行
modprobe br_netfilter -
加載ipvs 模塊
cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF chmod 755 /etc/sysconfig/modules/ipvs.modules sh /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_
3、etcd 集群部署
k8s集群使用堆疊 ETCD 時，可不單獨創建，本人是為了延長 etcd 證書才單獨部署集群，實際架構也是堆疊式
3.1 使用 etcdadm 工具
下載
# wget https://github.com/kubernetes-sigs/etcdadm/releases/download/v0.1.3/etcdadm-linux-amd64
# mv etcdadm-linux-amd64 /usr/local/bin/etcdadm
# chmod +x /usr/local/bin/etcdadm
選擇一個節點初始化etcd
etcdadm init --install-dir="/opt/etcd/" --name=etcd-1
參數解析
- --install-dir 安裝目錄
- --name 節點名稱
- server-cert-extra-sans 負載均衡地址（單獨部署使用）
拷貝證書到其他節點
scp /etc/etcd/pki/ca.* root@master2:/etc/etcd/pki/
scp /etc/etcd/pki/ca.* root@master3:/etc/etcd/pki/
加入集群
etcdadm join https://172.18.20.44:2379 --install-dir="/opt/etcd/" --name=etcd-2
安裝完成后查看集群
修改環境變量
source /etc/etcd/etcdctl.env
查看
# /opt/bin/etcdctl member list
18ed9897779ce358, started, k8s-master1, https://172.18.20.50:2380, https://172.18.20.50:2379, false
3e04e89693dd5c26, started, k8s-master2, https://172.18.20.51:2380, https://172.18.20.51:2379, false
68562c67a29d6f06, started, k8s-master3, https://172.18.20.49:2380, https://172.18.20.49:2379, false
建議直接使用 etcdctl.sh
注意：etcdadm 創建的證書期限為一年。需要一年更換一次，這對生產環境來說很不友好。目前有兩個解決方案
1、二進制部署 etcd，使用openssl 生成證書
-
2、修改etcdadm源碼
下載源碼
git clone https://github.com/kubernetes-sigs/etcdadm.git
修改 certs/pkiutil/pki_helpers.go 文件
certificateValidity =time.Hour * 24 * 365 * 10 # 10年期限
const (
// PrivateKeyBlockType is a possible value for pem.Block.Type.
PrivateKeyBlockType = "PRIVATE KEY"
// PublicKeyBlockType is a possible value for pem.Block.Type.
PublicKeyBlockType = "PUBLIC KEY"
// CertificateBlockType is a possible value for pem.Block.Type.
CertificateBlockType = "CERTIFICATE"
// RSAPrivateKeyBlockType is a possible value for pem.Block.Type.
RSAPrivateKeyBlockType = "RSA PRIVATE KEY"
rsaKeySize = 2048
certificateValidity = time.Hour * 24 * 365 * 10
)
編譯（需要golang 1.15 以上版本）
make
編譯完成后使用 etcdadm 重新生成集群
3.2、二進制部署
太麻煩了，網上找資料吧。ectdadm非常nice
4、安裝 docker 和 containerd (所有節點)
1.20 版本模式使用 containerd ，可不用安裝docker
添加 yum 源
# yum install -y yum-utils
# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
安裝
yum install -y containerd.io
生成默認配置
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
添加私有鏡像倉庫
containerd 修改 config.toml 配置
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
# 阿里云私有鏡像倉庫
[plugins."io.containerd.grpc.v1.cri".cri.registry.mirrors."registry-vpc.cn-shenzhen.aliyuncs.com"]
endpoint = ["http://registry-vpc.cn-shenzhen.aliyuncs.com"]
[plugins."io.containerd.grpc.v1.cri".registry.auths]
[plugins."io.containerd.grpc.v1.cri".registry.auths."registry-vpc.cn-shenzhen.aliyuncs.com"]
username = "username"
password = "password
啟動
systemctl restart containerd
systemctl enable containerd
5、使用kubeadm部署高可用集群
5.1、安裝kubeadm
yum源
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
安裝
yum install -y kubeadm kubelet kubectl
無法使用kubeadm啟動kubelet時，需要修改 kubelet 使用 container
# vim /usr/lib/systemd/system/kubelet.services.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
5.2、初始化集群
kubeadm默認會創建一個堆疊式的etcd集群，并不需要另外部署
kubeadm init --control-plane-endpoint "k8s-master-slb:6443" --upload-certs --node-name "k8s-master1"
參數解析
- --control-plane-endpoint apiserver集群地址
- --upload-certs 證書
由于國內無法訪問google的鏡像地址，這里需要使用國內的地址來下載鏡像，下載完成后需要更新 tag 為 k8s.gcr.io
5.3、使用外部 etcd 初始化集群
創建 kubeadm 初始化配置文件
# 生成 kubeadm 默認初始化模板
kubeadm config print init-defaults > kubeadm-config.yaml
kubeadm-config 使用說明 https://blog.51cto.com/foxhound/2517491?source=dra
修改
cat > kubeadm-config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
localAPIEndpoint:
advertiseAddress: 192.168.200.125
bindPort: 6443
nodeRegistration:
kubeletExtraArgs:
runtime-cgroups: /system.slice/containerd.service
kubelet-cgroups: /systemd/system.slice
container-runtime: remote
#container-runtime-endpoint: unix:///var/run/docker.sock
container-runtime-endpoint: unix:///run/containerd/containerd.sock
cgroup-driver: systemd
#criSocket: /var/run/docker.sock
criSocket: /run/containerd/containerd.sock
name: k8s-master1
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.20.4
networking:
dnsDomain: cluster.local
podSubnet: 10.10.0.0/16
serviceSubnet: 10.96.0.0/12
scheduler: {}
# 負載均衡地址
controlPlaneEndpoint: "k8s-master-slb:7443"
apiServer:
timeoutForControlPlane: 10m0s
certificatesDir: /etc/kubernetes/pki
clusterName: alw-cluster
controllerManager: {}
dns:
type: CoreDNS
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
etcd:
external:
endpoints:
- https://etcd-1:2379
- https://etcd-2:2379
- https://etcd-3:2379
caFile: /etc/etcd/pki/ca.crt
certFile: /etc/etcd/pki/apiserver-etcd-client.crt
keyFile: /etc/etcd/pki/apiserver-etcd-client.key
EOF
下載鏡像
kubeadm config images pull --config kubeadm-config.yaml
下載鏡像腳本
#!/bin/bash
images=(
kube-apiserver:v1.20.4
kube-controller-manager:v1.20.4
kube-scheduler:v1.20.4
kube-proxy:v1.20.4
pause:3.2
etcd:3.4.13-0
coredns:1.7.0)
for image in ${images[@]}; do
# docker
#docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/${image}
#docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/${image} k8s.gcr.io/${image}
#docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/${image}
# cri
ctr -n k8s.io i tag registry.cn-hangzhou.aliyuncs.com/google_containers/${image} k8s.gcr.io/${image}
done
初始化
kubeadm init --config kubeadm-config.yaml --upload-certs
初始化成功會出現以下信息
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join k8s-master-slb:7443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:138388af441152652765f8b1959e39db63c97049c3408a61c1a60cac5c8d8256 \
--control-plane --certificate-key bacab8cd43592812f0e3a186aaa615463c87e9280c0e2ae951b54b138325537d
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join k8s-master-slb:7443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:138388af441152652765f8b1959e39db63c97049c3408a61c1a60cac5c8d8256
然后集群二和集群三 運行命令加入
kubeadm join k8s-master-slb:7443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:138388af441152652765f8b1959e39db63c97049c3408a61c1a60cac5c8d8256 \
--control-plane --certificate-key bacab8cd43592812f0e3a186aaa615463c87e9280c0e2ae951b54b138325537d
查看集群
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master1 NotReady control-plane,master 11m v1.20.5
k8s-master2 NotReady control-plane,master 8m54s v1.20.5
k8s-master3 NotReady control-plane,master 8m31s v1.20.5
由于還未部署網絡插件（cni）,狀態還是 NotReady
-
注意：使用阿里云 SLB 時，由于apiserver 還未啟動，所以此時 6443 端口并未監聽，使用 SLB地址時會無法安裝成功。修改成當前服務器地址安裝成功后再切換 HOST 地址為 SLB地址
本機地址 172.18.20.44 負載均衡地址： 172.18.20.55 172.18.20.44 k8s-master-slb 注意：token有效期為24小時，失效后請在主節點使用以下命令重新生成
kubeadm token create --print-join-command
5.4、部署工作節點
安裝 containerd ,kubeadm,kubelet,kubectl 等
工作節點運行 join
kubeadm join k8s-master-slb:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:12b4fe0053bafd6b1b0e05482912b44ddcf88d1d1429e3c611d109ad5bf93ac0
master 查看節點
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready control-plane,master 6m28s v1.20.5
k8s-master2 NotReady control-plane,master 3m25s v1.20.5
k8s-master3 NotReady control-plane,master 3m11s v1.20.5
k8s-worker1 NotReady <none> 2m28s v1.20.5
k8s-worker2 NotReady <none> 4s v1.20.5
由于沒有部署 CNI 網絡插件，狀態還是 NotReady
5.5、部署 CNI 網絡插件
master 節點運行
需確保kube-flannel.yml文件里的 "Network": "10.10.0.0/16"IP內容與 kube-controller-manager.conf 配置的 --cluster-cidr 一致
即 kubeadm-config 里的 podSubnet
# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# kubectl apply -f kube-flannel.yml
# kubectl get pods -n kube-system
再次查看節點
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready control-plane,master 18h v1.20.5
k8s-master2 Ready control-plane,master 18h v1.20.5
k8s-master3 Ready control-plane,master 18h v1.20.5
k8s-worker1 Ready <none> 18h v1.20.5
k8s-worker2 Ready <none> 18h v1.20.5
5.6、Kubelet驅逐策略優化
修改工作節點kubelet啟動參數，更改Pod驅逐策略
vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="EVICTION_HARD=--eviction-hard=memory.available<2Gi,nodefs.available<5Gi,imagefs.available<100Gi"
Environment="EVICTION_RECLAIM=--eviction-minimum-reclaim=memory.available=0Mi,nodefs.available=1Gi,imagefs.available=2Gi"
重啟kubelet容器，并查看kubelet進程啟動參數
systemctl daemon-reload && systemctl restart kubelet
6、單獨部署coredns
不依賴kubeadm的方式，適用于不是使用kubeadm創建的k8s集群，或者kubeadm初始化集群之后，刪除了dns相關部署。
# 在calico網絡中也配置一個coredns # 10.96.0.10 為k8s官方指定的kube-dns地址
mkdir coredns && cd coredns
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh
chmod +x deploy.sh
./deploy.sh -i 10.96.0.10 > coredns.yml
kubectl apply -f coredns.yml
# 查看
kubectl get pods --namespace kube-system
kubectl get svc --namespace kube-system
7、Kubernetes 云管理控制器
自建 k8s 想要使用阿里云的負載均衡、存儲等服務時，需要部署阿里云提供的組件 cloud-controller-manager
github：https://github.com/kubernetes/cloud-provider-alibaba-cloud
幫助文檔：https://github.com/kubernetes/cloud-provider-alibaba-cloud/blob/master/docs/getting-started.md
7.1、安裝Alibaba CloudProvider 組件
修改 kubelet 啟動服務
修改kubelet 啟動參數，添加 --cloud-provider=external，并且在kubelet中添加
--hostname-override=${REGION_ID}.${INSTANCE_ID} --provider-id=${REGION_ID}.${INSTANCE_ID}
獲取 region-id 和 instance-id
echo `curl -s http://100.100.100.200/latest/meta-data/region-id`.`curl -s http://100.100.100.200/latest/meta-data/instance-id`
# vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--cloud-provider=external --hostname-override=cn-shenzhen.xxxxxxx --provider-id=cn-shenzhen.xxxxxxx
配置阿里云 AccessKeyID，AccessKeySecret
AccessKey & AccessKeySecret 必須以 base64 方式
# base64 AccessKey & AccessKeySecret
$ echo -n "$AccessKeyID" |base64
$ echo -n "$AcceessKeySecret"|base64
$ cat <<EOF >cloud-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: cloud-config
namespace: kube-system
data:
cloud-config.conf: |-
{
"Global": {
"accessKeyID": "$your-AccessKeyID-base64",
"accessKeySecret": "$your-AccessKeySecret-base64"
}
}
EOF
$ kubectl create -f cloud-config.yaml
添加 kubeconfig 配置文件(所有master節點)
vim /etc/kubernetes/cloud-controller-manager.conf
kind: Config
contexts:
- context:
cluster: alw-cluster
user: system:cloud-controller-manager
name: system:cloud-controller-manager@alw-cluster
current-context: system:cloud-controller-manager@alw-cluster
users:
- name: system:cloud-controller-manager
user:
tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: $CA_DATA
server: https://172.18.20.44:6443
name: alw-cluster
$CA_DATA 可以使用命令 cat /etc/kubernetes/pki/ca.crt|base64 -w 0 獲取。
server： 修改為本機IP
編寫 cloud-controller-manager.yaml 文件
也可以使用官方推薦的更詳細配置 cloud-controller-manager.yml
# vim cloud-controller-manager.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: cloud-controller-manager
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:cloud-controller-manager
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: cloud-controller-manager
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app: cloud-controller-manager
tier: control-plane
name: cloud-controller-manager
namespace: kube-system
spec:
selector:
matchLabels:
app: cloud-controller-manager
tier: control-plane
template:
metadata:
labels:
app: cloud-controller-manager
tier: control-plane
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
serviceAccountName: cloud-controller-manager
tolerations:
- effect: NoSchedule
operator: Exists
key: node-role.kubernetes.io/master
- effect: NoSchedule
operator: Exists
key: node.cloudprovider.kubernetes.io/uninitialized
nodeSelector:
node-role.kubernetes.io/master: ""
containers:
- command:
- /cloud-controller-manager
- --kubeconfig=/etc/kubernetes/cloud-controller-manager.conf
- --address=127.0.0.1
- --allow-untagged-cloud=true
- --leader-elect=true
- --cloud-provider=alicloud # Add your own cloud provider here!
- --use-service-account-credentials=true
- --cloud-config=/etc/kubernetes/config/cloud-config.conf
- --configure-cloud-routes=true
- --allocate-node-cidrs=true
- --route-reconciliation-period=3m
# replace ${cluster-cidr} with your own cluster cidr
- --cluster-cidr=10.10.0.0/16
image: registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3.339-g9830b58-aliyun
livenessProbe:
failureThreshold: 8
httpGet:
host: 127.0.0.1
path: /healthz
port: 10258
scheme: HTTP
initialDelaySeconds: 15
timeoutSeconds: 15
name: cloud-controller-manager
resources:
requests:
cpu: 200m
volumeMounts:
- mountPath: /etc/kubernetes/
name: k8s
- mountPath: /etc/ssl/certs
name: certs
- mountPath: /etc/pki
name: pki
- mountPath: /etc/kubernetes/config
name: cloud-config
hostNetwork: true
volumes:
- hostPath:
path: /etc/kubernetes
name: k8s
- hostPath:
path: /etc/ssl/certs
name: certs
- hostPath:
path: /etc/pki
name: pki
- configMap:
defaultMode: 420
items:
- key: cloud-config.conf
path: cloud-config.conf
name: cloud-config
name: cloud-config
配置解析
- --cloud-provider=alicloud 云服務商
- --cluster-cidr 集群 pod 地址
運行
# kubectl apply -f cloud-controller-manager.yaml
# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
cloud-controller-manager-7jbzc 1/1 Running 0 109m
cloud-controller-manager-dfpkv 1/1 Running 0 109m
cloud-controller-manager-lqvtz 1/1 Running 4 109m
完成組件的部署后，接下來就可以使用阿里云的負載均衡了
8、部署 Ingress-nginx
安裝 ingress-nginx 控制器
下載：
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.45.0/deploy/static/provider/cloud/deploy.yaml
國內無法訪問 google 的鏡像倉庫，需自己推一個到阿里云鏡像倉庫并設置為公有
# docker pull k8s.gcr.io/ingress-nginx/controller:v0.45.0
# docker tag k8s.gcr.io/ingress-nginx/controller:v0.45.0 registry.cn-shenzhen.aliyuncs.com/anlewo/ingress-nginx-controller:v0.45.0
# docker login --username=**** --password=**** registry.cn-shenzhen.aliyuncs.com
# docker push registry.cn-shenzhen.aliyuncs.com/anlewo/ingress-nginx-controller:v0.45.0
修改配置
...
apiVersion: apps/v1
kind: Deployment
……
# 國內無法訪問 google 的鏡像倉庫，需自己推一個到阿里云鏡像倉庫并設置為公有
image: registry-vpc.cn-shenzhen.aliyuncs.com/anlewo/ingress-nginx-controller:v0.45.0
……
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
helm.sh/chart: ingress-nginx-3.27.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.45.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
annotations:
# 指明SLB實例地址類型為私網類型
# service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: intranet
# 修改為您的私網SLB實例ID
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: *********-cn-shenzhen-st3-a01
# 是否自動創建SLB端口監聽（會覆寫已有端口監聽），也可手動創建端口監聽
#service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners: 'true'
spec:
type: LoadBalancer
externalTrafficPolicy: Local
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
運行
# kubectl apply -f deploy.yaml
9、部署Aliyun存儲插件 csi
9.1、官方ack csi插件
阿里云官方文檔：https://help.aliyun.com/document_detail/134722.html?spm=a2c4g.11186623.6.822.7c525ccfsHWlPe
github：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver
RBAC 安裝：
下載RBAC配置文件到操作機，并部署：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/deploy/rbac.yaml
執行：
$ kubectl apply -f rbac.yaml
CSI-Plugin 安裝：
下載普通模版
1. 下載模板：
下載最新版本的CSI Plugin部署模板：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/deploy/ack/csi-plugin.yaml
將部署模板下載到您的操作機，并保存（csi-plugin.yaml）。
2. 適配模板并部署：
根據集群所在的Region修改模板中的鏡像地址。例如：如果是cn-shenzhen的集群
則將 registry.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v1.2.0 中的：
registry 改為 registry-vpc
cn-hangzhou 改為 cn-shenzhen
即：registry-vpc.cn-shenzhen.aliyuncs.com/acs/csi-node-driver-registrar:v1.2.0
模板中的其他鏡像也是如此更新；
執行部署：
$ kubectl apply -f csi-plugin.yaml
3. 檢查安裝情況：
$ kubectl get pod -nkube-system | grep csi-plugin
$ kubectl describe ds csi-plugin -nkube-system | grep Image
CSI-Provisioner 安裝：
1. 下載模板：
下載最新版本的CSI Provisioner部署模板：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/deploy/ack/csi-provisioner.yaml
將部署模板下載到您的操作機，并保存（csi-provisioner.yaml）。
2. 適配模板并部署：
根據集群所在的Region修改模板中的鏡像地址。例如：如果是cn-beijing的集群
則將 registry.cn-hangzhou.aliyuncs.com/acs/csi-provisioner:v1.6.0-e360c7e43-aliyun 中的：
registry 改為 registry-vpc
cn-hangzhou 改為 cn-shenzhen
即：registry-vpc.cn-shenzhen.aliyuncs.com/acs/csi-provisioner:v1.6.0-e360c7e43-aliyun
模板中的其他鏡像也是如此更新；
執行部署：
$ kubectl apply -f csi-provisioner.yaml
3. 檢查安裝情況：
$ kubectl get pod -nkube-system | grep csi-provisioner
$ kubectl describe deploy csi-provisioner -nkube-system | grep Image
10、部署Dashboard
下載部署文件：
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.0/aio/deploy/recommended.yaml
默認Dashboard只能集群內部訪問，修改Service為NodePort類型，暴露到外部：
# vim recommended.yaml
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
ports:
- port: 443
targetPort: 8443
nodePort: 30001
type: NodePort
selector:
k8s-app: kubernetes-dashboard
部署：
# kubectl apply -f recommended.yaml
# kubectl get pods,svc -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
pod/dashboard-metrics-scraper-6b4884c9d5-mjl66 1/1 Running 0 23h
pod/kubernetes-dashboard-7bfbb48676-frmsf 1/1 Running 0 23h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dashboard-metrics-scraper ClusterIP 10.0.0.173 <none> 8000/TCP 23h
service/kubernetes-dashboard NodePort 10.0.0.145 <none> 443:30001/TCP 23h
訪問地址：https://NodeIP:30001
創建service account并綁定默認cluster-admin管理員集群角色：
kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
查看 token：
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
使用輸出的 token 登錄 Dashboard
結語
至此，一個生成環境可用的，基于阿里云ecs的k8s集群就搭建完成。但集群的搭建只是基礎，后續的維護使用才是重點，包括prometheus監控，istio等
