一. 官方提供的三種部署方式
minikube
Minikube是一個(gè)工具，可以在本地快速運(yùn)行一個(gè)單點(diǎn)的Kubernetes，僅用于嘗試Kubernetes或日常開發(fā)的用戶使用。
部署地址：https://kubernetes.io/docs/setup/minikube/
kubeadm
Kubeadm也是一個(gè)工具，提供kubeadm init和kubeadm join，用于快速部署Kubernetes集群。
部署地址：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/
二進(jìn)制包
推薦，從官方下載發(fā)行版的二進(jìn)制包，手動(dòng)部署每個(gè)組件，組成Kubernetes集群。
下載地址：https://github.com/kubernetes/kubernetes/releases/
二. Kubernetes平臺(tái)環(huán)境規(guī)劃
1.組件版本
| 軟件 | 版本 |
|---|---|
| linux 系統(tǒng) | centos7.6_x64 |
| Kubernetes | 1.12 |
| Docker | 18.xx-ce |
| Etcd | 3.x |
| Flannel | 0.10 |
2. 角色節(jié)點(diǎn)IP組件
| 角色 | IP | 組件 | 推薦配置 |
|---|---|---|---|
| master01 | 10.40.6.201 | kube-apiserver kube-controller-manager kube-scheduler etcd |
2核4G+ |
| master02 | 10.40.6.209 | kube-apiserver kube-controller-manager kube-scheduler |
2核4G+ |
| node01 | 10.40.6.210 | kubelet kube-proxy docker flannel etcd |
2核4G+ |
| node02 | 10.40.6.213 | kubelet kube-proxy docker flannel etcd |
2核4G+ |
| Load Balancer(Master) | 10.40.6.166 10.40.6.175 (VIP) |
Nginx L4 | 2核4G+ |
| Load Balancer(Backup) | 10.40.6.167 | Nginx L4 | 2核4G+ |
| Registry | 10.40.6.214 | Harbor | 2核4G+ |
3. 集群架構(gòu)
單Master集群架構(gòu)圖.png
多Master集群架構(gòu)圖.png
三. k8s自簽SSL證書
部署前建議把selinux, firewalld,關(guān)閉，
將配置文件/etc/selinux/config參數(shù)改為SELINUX=disabled，即時(shí)生效setenforce 0
停止firewalld: systemctl stop firewalld
每臺(tái)主機(jī)修改為相應(yīng)的主機(jī)名
| 組件 | 使用的證書 |
|---|---|
| etcd | ca.pem，server.pem，server-key.pem |
| flannel | ca.pem，server.pem，server-key.pem |
| kube-apiserver | ca.pem，server.pem，server-key.pem |
| kubelet | ca.pem，ca-key.pem |
| kube-proxy | ca.pem，kube-proxy.pem，kube-proxy-key.pem |
| kubectl | ca.pem，admin.pem，admin-key.pem |
四. Etcd數(shù)據(jù)庫集群部署
?二進(jìn)制包下載地址
https://github.com/etcd-io/etcd/releases
| 角色 | IP | 組件 |
|---|---|---|
| k8s-master01 | 10.40.6.201 | kube-apiserver kube-controller-manager kube-scheduler etcd |
| k8s-node1 | 10.40.6.210 | kubelet kube-proxy docker flannel etcd |
| k8s-node2 | 10.40.6.213 | kubelet kube-proxy docker flannel etcd |
1. 安裝cfssl工具
使用cfssl來生成自簽證書，先下載cfssl工具:
# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
# chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
# mv cfssl_linux-amd64 /usr/local/bin/cfssl
# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
# mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
2. 生成etcd證書
創(chuàng)建以下三個(gè)文件：cd /usr/local/src/k8s/etcd-cert
# cat ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
# cat ca-csr.json
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
# cat server-csr.json
{
"CN": "etcd",
"hosts": [
"10.40.6.201",
"10.40.6.210",
"10.40.6.213"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
生成證書：
# cd /usr/local/src/k8s/etcd-cert
# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
# ls *pem
ca-key.pem ca.pem server-key.pem server.pem
3. 部署Etcd
二進(jìn)制包下載地址：https://github.com/coreos/etcd/releases/tag/v3.2.12
以下部署步驟在規(guī)劃的三個(gè)etcd節(jié)點(diǎn)操作一樣，唯一不同的是etcd配置文件中的IP和節(jié)點(diǎn)名
解壓二進(jìn)制包：
# mkdir /opt/etcd/{bin,cfg,ssl} -p
# tar xvf etcd-v3.3.10-linux-amd64.tar.gz
# mv etcd-v3.3.10-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
創(chuàng)建etcd配置文件：
# cat /opt/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.40.6.201:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.40.6.201:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.40.6.201:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.40.6.201:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.40.6.201:2380,etcd02=https://10.40.6.210:2380,etcd03=https://10.40.6.213:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_NAME 節(jié)點(diǎn)名稱
ETCD_DATA_DIR 數(shù)據(jù)目錄
ETCD_LISTEN_PEER_URLS 集群通信監(jiān)聽地址
ETCD_LISTEN_CLIENT_URLS 客戶端訪問監(jiān)聽地址
ETCD_INITIAL_ADVERTISE_PEER_URLS 集群通告地址
ETCD_ADVERTISE_CLIENT_URLS 客戶端通告地址
ETCD_INITIAL_CLUSTER 集群節(jié)點(diǎn)地址
ETCD_INITIAL_CLUSTER_TOKEN 集群Token
ETCD_INITIAL_CLUSTER_STATE 加入集群的當(dāng)前狀態(tài)，new是新集群，existing表示加入已有集群
systemd管理etcd：
# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd
ExecStart=/opt/etcd/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
證書拷貝至配置文件指定的位置：
# cp /usr/local/src/k8s/etcd-cert/{ca,server-key,server}.pem /opt/etcd/ssl/
# ls /opt/etcd/ssl/
ca.pem server-key.pem server.pem
啟動(dòng)并設(shè)置開啟啟動(dòng)：
# systemctl start etcd
Job for etcd.service failed because a timeout was exceeded. See "systemctl status etcd.service" and "journalctl -xe" for details.
查看日志可以發(fā)現(xiàn)其他兩個(gè)節(jié)點(diǎn)未加入集群
# systemctl enable etcd
其他連個(gè)節(jié)點(diǎn)部署：
將10.40.6.201節(jié)點(diǎn)的相關(guān)目錄文件cp到其他兩個(gè)節(jié)點(diǎn)，并修改etcd 配置文件即可：
/opt/etcd/cfg/etcd
# scp -r /opt/etcd 10.40.6.210:/opt/
# scp -r /opt/etcd 10.40.6.213:/opt/
# scp /usr/lib/systemd/system/etcd.service 10.40.6.210:/usr/lib/systemd/system/
# scp /usr/lib/systemd/system/etcd.service 10.40.6.213:/usr/lib/systemd/system/
部署完成后，檢查etcd集群狀態(tài)：
# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--endpoints="https://10.40.6.201:2379,https://10.40.6.210:2379,https://10.40.6.213:2379" cluster-health
member 11e9f13e775913c8 is healthy: got healthy result from https://10.40.6.213:2379
member 188c1664ca149fb2 is healthy: got healthy result from https://10.40.6.210:2379
member 1e3d872c12b243a1 is healthy: got healthy result from https://10.40.6.201:2379
cluster is healthy
如果輸出上面信息，就說明集群部署成功。如果有問題第一步先看日志：/var/log/message 或 journalctl -u etcd
五. Node節(jié)點(diǎn)安裝Docker
# yum install -y yum-utils device-mapper-persistent-data lvm2
# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# yum install docker-ce -y
# curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://bc437cce.m.daocloud.io
# systemctl start docker
# systemctl enable docker
六. Flannel容器集群網(wǎng)絡(luò)部署
1. K8S網(wǎng)絡(luò)模型(CNI)
Container Network Interface(CNI):容器網(wǎng)絡(luò)接口，Google和CoreOS主導(dǎo)
2. K8S網(wǎng)絡(luò)模型設(shè)計(jì)基本要求
① 一個(gè)Pod一個(gè)IP
② 每個(gè)Pod 獨(dú)立一個(gè)IP, Pod內(nèi)所有容器共享網(wǎng)絡(luò)(同一個(gè)IP)
③ 所有容器都可以與所有其他容器通信
④ 所有節(jié)點(diǎn)都可以與所有容器通信
3. K8S最常用網(wǎng)絡(luò)插件
flannel: 隧道方案，對(duì)數(shù)據(jù)做封裝然后在解封裝，較消耗性能，node在100臺(tái)機(jī)器以下較為推薦
Calico: 路由方案，通過路由表轉(zhuǎn)發(fā)，不用對(duì)數(shù)據(jù)包做封裝和解封裝，性能較好,node100臺(tái)以上
4.部署Kubernetes網(wǎng)絡(luò) Flannel
Overlay Network：覆蓋網(wǎng)絡(luò)，在基礎(chǔ)網(wǎng)絡(luò)上疊加的一種虛擬網(wǎng)絡(luò)技術(shù)模式，該網(wǎng)絡(luò)中的主機(jī)通過虛擬鏈路連接起來。
VXLAN：將源數(shù)據(jù)包封裝到UDP中，并使用基礎(chǔ)網(wǎng)絡(luò)的IP/MAC作為外層報(bào)文頭進(jìn)行封裝，然后在以太網(wǎng)上傳輸，到達(dá)目的地后由隧道端點(diǎn)解封裝并將數(shù)據(jù)發(fā)送給目 標(biāo)地址。
Flannel：是Overlay網(wǎng)絡(luò)的一種，也是將源數(shù)據(jù)包封裝在另一種網(wǎng)絡(luò)包里面進(jìn)行路由轉(zhuǎn)發(fā)和通信，目前已經(jīng)支持UDP、VXLAN、Host-GW、AWS VPC和GCE路由等數(shù)據(jù)轉(zhuǎn)發(fā)方式。
node節(jié)點(diǎn)都在一個(gè)局域網(wǎng)內(nèi)建議使用 Host-GW，性能幾乎沒有損耗
node節(jié)點(diǎn)跨網(wǎng)段，建議使用 VXLAN，對(duì)基礎(chǔ)網(wǎng)絡(luò)環(huán)境比較嚴(yán)格，只要在任何互聯(lián)網(wǎng)絡(luò)里，只要能通信就可以使用。
1). Flannel網(wǎng)絡(luò)工作原理
Overlay Network.png
flannel工作原理.png
2). 分配子網(wǎng)段寫入etcd
分配子網(wǎng)段寫入etcd供flanneld使用:
# /opt/etcd/bin/etcdctl \
--ca-file=/opt/etcd/ssl/ca.pem \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--endpoints="https://10.40.6.201:2379,https://10.40.6.210:2379,https://10.40.6.213:2379" \
set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}
給/coreos.com/network/config key 劃分一個(gè)大的子網(wǎng)172.17.0.0/16，類型為 vxlan。
可以通過get 獲取這個(gè)/coreos.com/network/config key的值
# /opt/etcd/bin/etcdctl \
--ca-file=/opt/etcd/ssl/ca.pem \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--endpoints="https://10.40.6.201:2379,https://10.40.6.210:2379,https://10.40.6.213:2379" \
get /coreos.com/network/config
3). 下載二進(jìn)制包
https://github.com/coreos/flannel/releases
# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
# tar xvf flannel-v0.11.0-linux-amd64.tar.gz
# mkdir /opt/kubernetes/bin -p
# mv flanneld mk-docker-opts.sh /opt/kubernetes/bin
4). 部署與配置Flannel
# mkdir /opt/kubernetes/cfg -p
# cat /opt/kubernetes/cfg/flanneld
FLANNEL_OPTIONS="--etcd-endpoints=https://10.40.60.201:2379,https://10.40.60.210:2379,https://10.40.6.213:2379 -etcd-cafile=/opt/etcd/ssl/ca.pem -etcd-certfile=/opt/etcd/ssl/server.pem -etcd-keyfile=/opt/etcd/ssl/server-key.pem"
5). systemd管理Flannel
flannel 啟動(dòng)后配置的子網(wǎng)保存到/run/flannel/subnet.env文件中
# cat /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
[Install]
WantedBy=multi-user.target
6). 配置Docker啟動(dòng)使用Flannel生成的子網(wǎng)
docker啟動(dòng)時(shí)讀取flannel子網(wǎng)文件 /run/flannel/subnet.env
# cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
7). 重啟flannel和docker
# systemctl daemon-reload
# systemctl start flanneld
# systemctl enable flanneld
# systemctl restart docker
8). 檢查是否生效
# ps -ef |grep docker
root 17311 1 0 16:07 ? 00:00:00 /usr/bin/dockerd --bip=172.17.31.1/24 --ip-masq=false --mtu=1450
# ip addr
.......
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:94:ca:12:8a brd ff:ff:ff:ff:ff:ff
inet 172.17.31.1/24 brd 172.17.31.255 scope global docker0
valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 16:c8:15:51:0c:30 brd ff:ff:ff:ff:ff:ff
inet 172.17.31.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::14c8:15ff:fe51:c30/64 scope link
valid_lft forever preferred_lft forever
確保docker0與flannel.1在同一網(wǎng)段。
測試不同節(jié)點(diǎn)互通，在當(dāng)前節(jié)點(diǎn)訪問另一個(gè)Node節(jié)點(diǎn)docker0 IP：
# ping -c 2 172.17.59.1
PING 172.17.59.1 (172.17.59.1) 56(84) bytes of data.
64 bytes from 172.17.59.1: icmp_seq=1 ttl=64 time=0.355 ms
64 bytes from 172.17.59.1: icmp_seq=2 ttl=64 time=0.293 ms
如果能通說明Flannel部署成功。如果不通檢查下日志：journalctl -u flannel
啟動(dòng)一個(gè)容器，在另一個(gè)node節(jié)點(diǎn)ping 容器的IP，兩個(gè)節(jié)點(diǎn)都啟動(dòng)一個(gè)容器，容器里互ping，測試容器是否互通
# docker run -it busybox
9). 獲取Etcd中的Flannel網(wǎng)絡(luò)信息
列出子網(wǎng)父目錄:
# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://10.40.6.201:2379,https://10.40.6.210:2379,https://10.40.6.213:2379" ls /coreos.com/network/
/coreos.com/network/config
/coreos.com/network/subnets
# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://10.40.6.201:2379,https://10.40.6.210:2379,https://10.40.6.213:2379" ls /coreos.com/network/subnets
/coreos.com/network/subnets/172.17.31.0-24
/coreos.com/network/subnets/172.17.59.0-24
# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://10.40.6.201:2379,https://10.40.6.210:2379,https://10.40.6.213:2379" get /coreos.com/network/subnets/172.17.31.0-24
{"PublicIP":"10.40.6.210","BackendType":"vxlan","BackendData":{"VtepMAC":"16:c8:15:51:0c:30"}}
/coreos.com/network/config ：分配的子網(wǎng)存儲(chǔ)key
/coreos.com/network/subnets ：分配node節(jié)點(diǎn)的子網(wǎng)存儲(chǔ)key父目錄,key名標(biāo)識(shí)了節(jié)點(diǎn)給容器分配的子網(wǎng)
七. 部署Master組件
在部署Kubernetes之前一定要確保etcd、flannel、docker是正常工作的，否則先解決問題再繼續(xù)
第三個(gè)組件：
kube-apiserver
kube-controller-manager
kube-scheduler
步驟： 配置文件 -> systemd管理組件 -> 啟動(dòng)
1. 生成證書
創(chuàng)建CA證書：
# cat ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
# cat ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "System"
}
]
}
# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
生成apiserver證書：
下面的IP主要是master和LB的IP， 第一個(gè)IP是service
# cat server-csr.json
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"10.40.6.201",
"10.40.6.209",
"10.40.6.166",
"10.40.6.175",
"10.40.6.167",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
生成kube-proxy證書：
# cat kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
最終生成以下證書文件：
# ls *pem
ca-key.pem ca.pem kube-proxy-key.pem kube-proxy.pem server-key.pem server.pem
拷貝證書到證書目錄：/opt/kubernetes/ssl/
# cp ca.pem server.pem server-key.pem ca-key.pem /opt/kubernetes/ssl/
2. 部署apiserver組件
下載二進(jìn)制包：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md
下載這個(gè)包（kubernetes-server-linux-amd64.tar.gz）就夠了，包含了所需的所有組件。
# mkdir /opt/kubernetes/{bin,cfg,ssl} -p
# tar xvf kubernetes-server-linux-amd64.tar.gz
# cd kubernetes/server/bin
# cp kube-apiserver kube-scheduler kube-controller-manager kubectl /opt/kubernetes/bin
創(chuàng)建token文件，用于kubelet請(qǐng)求簽名（請(qǐng)求加入集群時(shí)頒發(fā)證書使用）：
# head -c 16 /dev/urandom |od -An -t x |tr -d ' ' ##生成 token id
5b2ecab909e3ae8f0dc611ba255777c2
# cat /opt/kubernetes/cfg/token.csv
5b2ecab909e3ae8f0dc611ba255777c2,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
第一列：隨機(jī)字符串，自己可生成
第二列：用戶名
第三列：UID
第四列：用戶組,kubernetes的一個(gè)用戶角色
創(chuàng)建apiserver配置文件：
# cat /opt/kubernetes/cfg/kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=false \
--log-dir=/opt/kubernetes/logs/kube-apiserver \
--v=4 \
--etcd-servers=https://10.40.6.201:2379,https://10.40.6.210:2379,https://10.40.6.213:2379 \
--bind-address=10.40.6.201 \
--secure-port=6443 \
--advertise-address=10.40.6.201 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--service-node-port-range=30000-50000 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"
配置好前面生成的證書，確保能連接etcd。
參數(shù)說明：
--logtostderr 啟用日志,true日志將會(huì)寫到/var/log/messages,不用指定log日志路徑；false，自定義日志文件
---v 日志等級(jí)，值越大，日志越少
--etcd-servers etcd集群地址
--bind-address 監(jiān)聽地址
--secure-port https安全端口
--advertise-address 集群通告地址
--allow-privileged 啟用授權(quán)，容器層面的
--service-cluster-ip-range Service負(fù)責(zé)均衡的虛擬IP地址段
--service-node-port-range Service Node類型默認(rèn)分配端口范圍
--enable-admission-plugins 準(zhǔn)入控制模塊（插件）
--authorization-mode 認(rèn)證授權(quán)模式，啟用RBAC授權(quán)和節(jié)點(diǎn)自管理
--enable-bootstrap-token-auth 啟用TLS bootstrap功能，用于驗(yàn)證kubelet發(fā)過來的請(qǐng)求，給kubeltet頒發(fā)證書，如node加入集群等
--token-auth-file token文件
創(chuàng)建日志目錄：
# mkdir /opt/kubernetes/logs/kube-apiserver -p
systemd管理apiserver：
# cat /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
啟動(dòng)：
# systemctl daemon-reload
# systemctl enable kube-apiserver
# systemctl restart kube-apiserver
查看進(jìn)程：ps aux |grep kube
3. 部署scheduler組件
創(chuàng)建schduler配置文件：
# cat /opt/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--log-dir=/opt/kubernetes/logs/kube-scheduler \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect"
參數(shù)說明：
--master 連接本地apiserver
--leader-elect 當(dāng)該組件啟動(dòng)多個(gè)時(shí)，自動(dòng)選舉（HA）
創(chuàng)建日志目錄：
# mkdir /opt/kubernetes/logs/kube-scheduler -p
systemd管理schduler組件：
# cat /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
啟動(dòng)：
# systemctl daemon-reload
# systemctl enable kube-scheduler
# systemctl restart kube-scheduler
4. 部署controller-manager組件
創(chuàng)建controller-manager配置文件：
# cat /opt/kubernetes/cfg/kube-controller-manager
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--log-dir=/opt/kubernetes/logs/kube-controller-manager \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect=true \
--address=127.0.0.1 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--experimental-cluster-signing-duration=87600h0m0s"
參數(shù)說明：
--master kube-apiserver監(jiān)聽地址
--leader-elect 集群角色選舉
--address 監(jiān)聽地址，controller-manager不對(duì)外服務(wù)
--cluster-name 集群名字
--cluster-signing-cert-file 簽名，為給kubelet頒發(fā)證書使用
--cluster-signing-key-file 簽名，為給kubelet頒發(fā)證書使用
--root-ca-file 簽名，為給kubelet頒發(fā)證書使用
--service-account-private-key-file 簽名，為給kubelet頒發(fā)證書使用
--experimental-cluster-signing-duration=87600h0m0s 給kubelet頒發(fā)證書時(shí)間，默認(rèn)一年
創(chuàng)建日志目錄：
# mkdir /opt/kubernetes/logs/kube-controller-manager -p
systemd管理controller-manager組件：
# cat /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
啟動(dòng)：
# systemctl daemon-reload
# systemctl enable kube-controller-manager
# systemctl restart kube-controller-manager
所有組件都已經(jīng)啟動(dòng)成功，通過kubectl工具查看當(dāng)前集群組件狀態(tài)：
# cp /usr/local/src/k8s/kubernetes/server/bin/kubectl /usr/bin/
# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
如上輸出說明組件都正常。
查看資源的縮寫：kubectl api-resources
八. 部署Node組件
Master apiserver啟用TLS認(rèn)證后，Node節(jié)點(diǎn)kubelet組件想要加入集群，必須使用CA簽發(fā)的有效證書才能與apiserver通信，當(dāng)Node節(jié)點(diǎn)很多時(shí)，簽署證書是一件很繁瑣的事情，因此有了TLS Bootstrapping機(jī)制，kubelet會(huì)以一個(gè)低權(quán)限用戶自動(dòng)向apiserver申請(qǐng)證書，kubelet的證書由apiserver動(dòng)態(tài)簽署。
認(rèn)證大致工作流程如圖所示:
工作流程.png
master使用的token文件:
# cat /opt/kubernetes/cfg/token.csv
5b2ecab909e3ae8f0dc611ba255777c2,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
第一列：token ID
第二列：用戶名 kubelet-bootstrap
第三列：UID
第四列：用戶組,kubernetes的一個(gè)用戶角色
1. 角色與用戶綁定
創(chuàng)建clusterrolebinding kubelet-bootstrap并將用戶kubelet-bootstrap綁定到system:node-bootstrapper集群角色
# kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
2. 創(chuàng)建kubeconfig文件
生成bootstrap.kubeconfig和kube-proxy.kubeconfig文件腳本：
# cat kubeconfig.sh
APISERVER=$1
SSL_DIR=$2
#token值要與master文件/opt/kubernetes/cfg/token.csv 里的一致
BOOTSTRAP_TOKEN='5b2ecab909e3ae8f0dc611ba255777c2'
# 創(chuàng)建kubelet bootstrapping kubeconfig
export KUBE_APISERVER="https://$APISERVER:6443"
# 設(shè)置集群參數(shù)
kubectl config set-cluster kubernetes \
--certificate-authority=$SSL_DIR/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig
# 設(shè)置客戶端認(rèn)證參數(shù)
kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=bootstrap.kubeconfig
# 設(shè)置上下文參數(shù)
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig
# 設(shè)置默認(rèn)上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
#----------------------
# 創(chuàng)建kube-proxy kubeconfig文件,存放連接apiserver驗(yàn)證信息
kubectl config set-cluster kubernetes \
--certificate-authority=$SSL_DIR/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
--client-certificate=$SSL_DIR/kube-proxy.pem \
--client-key=$SSL_DIR/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
傳入apiserver IP 和生成kubernetes證書的目錄兩個(gè)參數(shù)，運(yùn)行腳本生成bootstrap.kubeconfig和kube-proxy.kubeconfig兩個(gè)配置文件
# bash kubeconfig.sh 10.40.6.201 /usr/local/src/k8s/kube-apiserver
# cat bootstrap.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVVVRhRFdja1dBT2xLd2s3S0ZMNjFTb0xkUmpJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU1TURVeU9URTFNVGt3TUZvWERUSTBNRFV5TnpFMU1Ua3dNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbGFXcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBd3BFUW5hZCtkbkFmamVhbzNMVUIKRWdvWWN4ZFNUTjdkS0FrV2NxNkY3ZVZWVUR1RmZkWFc2VWdCR3RqeUpoTEhKREF1a01wc2gzSUw2cW95U0lraQoyTGdmTVFTOFhEQmhRUXY5OFhRVnVvMG44dVhzQ08yZjdpS2hpM3NUb0VIWTJGVmNYM1BUOTgvN1A4cTBpZzArCm84RjBrNXVaTzJjT1hIWDF0c0NLL3FrMWp1S3J3Wk04enpDRUszbGZJNmtROUltT01NZG93MHE0bzZEdStPWVAKaUE2MFVHRnhpd0VlTWs1b2JBN2liZ1BSai81ci9BSVdZbmUvV0Y3ODM5bW1kY2ZUVXRJdzlJbHd4eEN3MEV1dQo0NXkwcVAwL3RJNUZlSFA4VWJzRWdCOFJuWTArTG03TXk1Z2lDYSt4cTNQUlh0UVozRDBwYm8xYXFuakh6NnFqCkpRSURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVeDU2TkxhajVwNXliTEJ4M1k5VmR6cldmSjF3d0h3WURWUjBqQkJnd0ZvQVV4NTZOTGFqNQpwNXliTEJ4M1k5VmR6cldmSjF3d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFNQXBoYW14a2tRRUQvYk80NlVmCkdCWVBtQjZMZy9WcWx0MlBxaFFWblEwRmMySldtRzFlY2l0Q1JLWVlYU2RPRkZjNXorcWlCZjVDQWFRVm5vWnUKNTB3QTBmajBqb3BSbnRCWDR6YzRXdmM0WEZWYjVKRXZFcjRqOEpkTTI1QXhHN0hsazA4RzRRbEZ3SzNuRVo4dwptMGtjaEJpb2tFZElmWEZvekttWThxUTNmY0o4MEVONmJBYXJHQVNoK2VQTXVxMXhqNjhwUVJBSnArcFNyMVNHCnFIamxhbnpRT1hSeitMZFBhSXQzQjEzMDFsSyt1ZnlZbHVGcGJ1c24ycmlXOHlmMXhyeEhRb0VyTTllZTMvR0sKUnZVMnc0WGpwVEdXbGNwMU1uNVA4MGNDVlQwa05VS1BBWVVzaFF6RlNITlhYU2lsZEpQNG80RnhIQ3JzMGxoRApqL2c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
server: https://10.40.6.201:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubelet-bootstrap
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
user:
token: 5b2ecab909e3ae8f0dc611ba255777c2
# cat kube-proxy.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVVVRhRFdja1dBT2xLd2s3S0ZMNjFTb0xkUmpJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU1TURVeU9URTFNVGt3TUZvWERUSTBNRFV5TnpFMU1Ua3dNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbGFXcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBd3BFUW5hZCtkbkFmamVhbzNMVUIKRWdvWWN4ZFNUTjdkS0FrV2NxNkY3ZVZWVUR1RmZkWFc2VWdCR3RqeUpoTEhKREF1a01wc2gzSUw2cW95U0lraQoyTGdmTVFTOFhEQmhRUXY5OFhRVnVvMG44dVhzQ08yZjdpS2hpM3NUb0VIWTJGVmNYM1BUOTgvN1A4cTBpZzArCm84RjBrNXVaTzJjT1hIWDF0c0NLL3FrMWp1S3J3Wk04enpDRUszbGZJNmtROUltT01NZG93MHE0bzZEdStPWVAKaUE2MFVHRnhpd0VlTWs1b2JBN2liZ1BSai81ci9BSVdZbmUvV0Y3ODM5bW1kY2ZUVXRJdzlJbHd4eEN3MEV1dQo0NXkwcVAwL3RJNUZlSFA4VWJzRWdCOFJuWTArTG03TXk1Z2lDYSt4cTNQUlh0UVozRDBwYm8xYXFuakh6NnFqCkpRSURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVeDU2TkxhajVwNXliTEJ4M1k5VmR6cldmSjF3d0h3WURWUjBqQkJnd0ZvQVV4NTZOTGFqNQpwNXliTEJ4M1k5VmR6cldmSjF3d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFNQXBoYW14a2tRRUQvYk80NlVmCkdCWVBtQjZMZy9WcWx0MlBxaFFWblEwRmMySldtRzFlY2l0Q1JLWVlYU2RPRkZjNXorcWlCZjVDQWFRVm5vWnUKNTB3QTBmajBqb3BSbnRCWDR6YzRXdmM0WEZWYjVKRXZFcjRqOEpkTTI1QXhHN0hsazA4RzRRbEZ3SzNuRVo4dwptMGtjaEJpb2tFZElmWEZvekttWThxUTNmY0o4MEVONmJBYXJHQVNoK2VQTXVxMXhqNjhwUVJBSnArcFNyMVNHCnFIamxhbnpRT1hSeitMZFBhSXQzQjEzMDFsSyt1ZnlZbHVGcGJ1c24ycmlXOHlmMXhyeEhRb0VyTTllZTMvR0sKUnZVMnc0WGpwVEdXbGNwMU1uNVA4MGNDVlQwa05VS1BBWVVzaFF6RlNITlhYU2lsZEpQNG80RnhIQ3JzMGxoRApqL2c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
server: https://10.40.6.201:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kube-proxy
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kube-proxy
user:
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQzakNDQXNhZ0F3SUJBZ0lVVHNyZk1EeGZOVmlMb2o0RjN5NEkwQjRLT1ZFd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU1TURVek1EQXhOVGt3TUZvWERUSTVNRFV5TnpBeE5Ua3dNRm93YkRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUm93R0FZRFZRUURFeEZ6ZVhOMFpXMDZhM1ZpClpTMXdjbTk0ZVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS2RjS2hWWHRQb1AKRDJHMXRQQnFRamU4bFdRTmo1U1NhZjlSQ0xESld3bDJNSm1JVjkyclpEQVE5S2dRNnFkTlZQODFKYVB1Zy8xaQpONmtSdEl0VElYSFVDQ2RSRzdjUWZNZDEyV081cE9LLy82dnpaSTBsWXJIdEk4QUFHNU5UcU9FYmJYT3NyNkxaCjRodWh1T3h2elZLUWQxeUxBVHRZK1RCTU9xdEFweUp0dzRLdGxZNDVMTXZzTzdJUWZsWCtuQ0h2V1JFeFBjSVMKb0RxYlA0UHAzdFZQNDVZd0xYUVRWRFl1MitRU0VPUUwrMHJTbDFEcGdNbHk1KzRKSVR5VmRnLzJKM0tTaWpPRApKUFUzLzdMaGJmMEJXMDdvenowSkdRMzBFWUhEeFBRQUlEZlNqd0N4U0hwaEhubUVGN0FtRlY0clJPNHhmOVQvCmh3L21Pd1o0QXJrQ0F3RUFBYU4vTUgwd0RnWURWUjBQQVFIL0JBUURBZ1dnTUIwR0ExVWRKUVFXTUJRR0NDc0cKQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFNQmdOVkhSTUJBZjhFQWpBQU1CMEdBMVVkRGdRV0JCU2M1YmRqaGRxNApLVHJxRFk5a1NTNDNET0V2Y3pBZkJnTlZIU01FR0RBV2dCVEhubzB0cVBtbm5Kc3NISGRqMVYzT3RaOG5YREFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQUxCNXorNlUxbUVXdEZqeUVsemtPa2hyZzRUK0VqbG9BdkRFVFRZdEsKekxwY3dJU2ZseFg5Mk5FL3NBazhtVjZjdUhaNWNJQ0gyWjhSWmwvL3VJVWppT2pDYlpjcUxaYUl1cUlhUDE0NApaUnp2VmhMYkhyZXFQSit1bHdwZXExaEJydUgwK0pZeEhqN3VDbGxKWldRUFhXejNzR29od29MVWFSRXd2UlNVCnZ0OXk1dGRWNzJBOTdnSERQSGJDdFcySlhyR3NUSll3UndjcHNSbkg1QjVjelFNTkFrei9pUXdzUmN0ViszcXcKNFRZWkZWbTN0OTEwUTh0WDVtOXRGVTdOME1ydlZQL1FQSS9Idlp4ZEI2Q1R3cTVodnVaMVY2ck1mTWowVkZ1bwo5OE14ZFFBVUQxeVBqeldXMllQTEJ4enZFY09sQmdNVERmME03WVNaWDBsZG5RPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBcDF3cUZWZTArZzhQWWJXMDhHcENON3lWWkEyUGxKSnAvMUVJc01sYkNYWXdtWWhYCjNhdGtNQkQwcUJEcXAwMVUvelVsbys2RC9XSTNxUkcwaTFNaGNkUUlKMUVidHhCOHgzWFpZN21rNHIvL3EvTmsKalNWaXNlMGp3QUFiazFPbzRSdHRjNnl2b3RuaUc2RzQ3Ry9OVXBCM1hJc0JPMWo1TUV3NnEwQ25JbTNEZ3EyVgpqamtzeSt3N3NoQitWZjZjSWU5WkVURTl3aEtnT3BzL2crbmUxVS9qbGpBdGRCTlVOaTdiNUJJUTVBdjdTdEtYClVPbUF5WExuN2draFBKVjJEL1luY3BLS000TWs5VGYvc3VGdC9RRmJUdWpQUFFrWkRmUVJnY1BFOUFBZ045S1AKQUxGSWVtRWVlWVFYc0NZVlhpdEU3akYvMVArSEQrWTdCbmdDdVFJREFRQUJBb0lCQUZsUjNmL3dCRjJrNWYrdQorN2VIN25sU3c2UlhmSGE5d2FhSytBbHFIWlVxSi92NUFYUUVBZitKUFJucGxXTGU2ZXNlMFV6eGdpNGNXanA0CmdaUU9OUDVNUEdISGJ2US83MmlBcEJvT1BVcnJUNmZVeWFodStJS2ZYb0lkVEpwUGZ3Vk5IeGdxWkw2VWJKRjAKdVg0dW1UVmtkdC9FTEU4aFNEVVhxZ1EyQ0QxZDNIRk5hbUtRMTFrNlBrSC9OZDRQN09aS0hGM244bnR1eWdNawpxald5Y3VSODgreFJFNC9OdzdKOXk3VXR4dEVPd1lVL05vS3M4T1RBMW1ZV2h1Z2o0dSt5bEZxU3JWcXVmblNtCkpVNHlKQStvWEdpY3ZJaE1ZUmxpdjNEYkRTSGg5VHFYVXV0eEhoWVpLbS93RmlTR1JaL0VYT3creUZHcmkwYWQKZmplbUFsRUNnWUVBMXAzQU8vTjYzazMxYUh2WndQYzdpbGN4MHpETlJQbzlibXZFa3BwNnZ0WWpCOWpVeDhaQQpnYmNmWVJua29KVVdlUnNVVUphODBxKzlLK2tJeVVmd1kwdTd6RTF6Y0hqUWJ6Zzl0TDFJQlFmZTlwaDRJRHROCm1NeHE1RnVIV0VNRlFIUU41OFdQT3ZRQWtVUFpYNDZFRTBKM3c2QXIyODhtZVIrSlE4bjVueDBDZ1lFQXg2R3MKU3ZsR2NVSGEvS3BWelk3Wmx6UHVKMDV1U2ZReS9FcU9QRnNzNGdNR3RVNGROUzBkWm5GUnBJOWRjZGJlMUdxZwptV0FTdXM4QmdDZytTdDA5WjdOSnJLMjVrQytzU0dIdVd3OFM0aFV1ZSt2MDBrNTNhblloeWNCcW4yOWl0WkxNCnBGZDNCKzU1akxuUExySTlrVFFjUjd4U3JXanhMYmVlc1VXa0UwMENnWUIvNURYQUJCSCtFNXJnanAxdXZtVysKeE1NdVJQQ3Q0Q2xuZWRVRVFBWlJYcTQxYU9NendWS0RlaXE2NUlFM3FHQmgvdDhXUHgxNnQ3c1ZSYU0wdnlmagpKQ2hmVVBBdjMrN2x1REFkV29abWFSQlhCdmplekRncmkvVk82N1ExeG9xRXBDUDlMOTl3bENNYWJjSkZqVm5yCldEcWlXdnFIM0dQaTNnWWdYV1hoaVFLQmdFeFIyOHVoOXpOUGFRZ1ZtczRHWWR0emlBWFE3MHNvcCtGYUkzeWgKb3N3Wk9oUlFjOHdqbmt6TzM5YVkxTEd6NHVhMGlRZDUrazhlMnNVREhhV0RaWGxpeXJUUWlkTzgxaEdxRnZVTApFejRKdVFhNVU1U2ZXUG9EaGJGYTlhaFViaGxhc1EvWFBITjAwVlZpcC9tRFBSUnBKcktxSmJXVUhEaE5MY2M2CkI1czFBb0dCQUxCYkMySFh2dENTSXRkdHVlZ2k1ZlA1SUtCYVUxbnB0NWkvZ01aYlpNQkJHQllUZmVuNTI0U1MKaVdMWnNKalpGdmFqS3Q4RktZSHVhU29ETWhQZVJ3cWlaaTlkZkp6WFlBOWZENEk5RytUc2dFTU9tR3JpZ3NnTwpuMExVMTYyRUhLM1ZvRDVTL0cxUFl2TG50S1ZNT29CWmdJZzdzOW9RZEl0R1ovbmdpNTlQCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
將這兩個(gè)文件拷貝到Node節(jié)點(diǎn)/opt/kubernetes/cfg目錄下：
# scp bootstrap.kubeconfig kube-proxy.kubeconfig 10.40.6.210:/opt/kubernetes/cfg/
# scp bootstrap.kubeconfig kube-proxy.kubeconfig 10.40.6.213:/opt/kubernetes/cfg/
3. 部署kubelet組件
前面在master下載的二進(jìn)制包中的kubelet和kube-proxy拷貝到node節(jié)點(diǎn)的/opt/kubernetes/bin目錄下：
# scp /usr/local/src/k8s/kubernetes/server/bin/{kube-proxy,kubelet} 10.40.6.210:/opt/kubernetes/bin/
# scp /usr/local/src/k8s/kubernetes/server/bin/{kube-proxy,kubelet} 10.40.6.213:/opt/kubernetes/bin/
創(chuàng)建kubelet配置文件：
# cat /opt/kubernetes/cfg/kubelet
KUBELET_OPTS="--logtostderr=false \
--log-dir=/opt/kubernetes/logs/kubelet \
--v=4 \
--hostname-override=10.40.6.210 \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet.config \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
參數(shù)說明：
--hostname-override 在集群中顯示的主機(jī)名
--kubeconfig 指定kubeconfig文件位置，會(huì)自動(dòng)生成
--bootstrap-kubeconfig 指定剛才生成的bootstrap.kubeconfig文件
--cert-dir 頒發(fā)證書存放位置
--pod-infra-container-image 管理Pod網(wǎng)絡(luò)的鏡像
/opt/kubernetes/cfg/kubelet.config文件配置自身信息，配置如下：
# cat /opt/kubernetes/cfg/kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 10.40.6.210
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.0.0.2"]
clusterDomain: cluster.local.
failSwapOn: false
authentication:
anonymous:
enabled: true
systemd管理kubelet組件：
# cat /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet
ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
KillMode=process
[Install]
WantedBy=multi-user.target
創(chuàng)建相關(guān)目錄：
# mkdir /opt/kubernetes/{ssl,logs/kubelet} -p
啟動(dòng)：
# systemctl daemon-reload
# systemctl enable kubelet
# systemctl restart kubelet
在Master審批Node加入集群：
啟動(dòng)后還沒加入到集群中，需要手動(dòng)允許該節(jié)點(diǎn)才可以。
在Master節(jié)點(diǎn)查看請(qǐng)求簽名的Node：
# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-N3b6ze5SPhItvld_iaByflG6tZn3mhUpyjxwOwTLdU4 4m4s kubelet-bootstrap Pending
# kubectl certificate approve node-csr-N3b6ze5SPhItvld_iaByflG6tZn3mhUpyjxwOwTLdU4
certificatesigningrequest.certificates.k8s.io/node-csr-N3b6ze5SPhItvld_iaByflG6tZn3mhUpyjxwOwTLdU4 approved
# kubectl get node
NAME STATUS ROLES AGE VERSION
10.40.6.210 Ready <none> 23s v1.12.1
部署第二個(gè)節(jié)點(diǎn)，修改配置文件中的相應(yīng)IP即可
# kubectl get node
NAME STATUS ROLES AGE VERSION
10.40.6.210 Ready <none> 6m33s v1.12.1
10.40.6.213 NotReady <none> 9s v1.12.1
4. 部署 kube-proxy組件
創(chuàng)建kube-proxy配置文件：
# cat /opt/kubernetes/cfg/kube-proxy
KUBE_PROXY_OPTS="--logtostderr=false \
--log-dir=/opt/kubernetes/logs/kube-proxy \
--v=4 \
--hostname-override=10.40.6.210 \
--cluster-cidr=10.0.0.0/24 \
--proxy-mode=ipvs \
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"
參數(shù)說明：
--cluster-cidr 分配集群的網(wǎng)段，service負(fù)載均衡IP段
--proxy-mode 代理模式
systemd管理kube-proxy組件：
# cat /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
創(chuàng)建相關(guān)目錄：
# mkdir /opt/kubernetes/logs/kube-proxy -p
啟動(dòng)：
# systemctl daemon-reload
# systemctl enable kube-proxy
# systemctl restart kube-proxy
Node2部署方式一樣, 修改配置文件相關(guān)IP
九. 部署一個(gè)測試示例
創(chuàng)建一個(gè)Nginx Web，測試集群是否正常工作：
# kubectl run nginx --image=nginx --replicas=3
# kubectl get deployment ###查看剛創(chuàng)建的nginx deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
nginx 3 3 3 3 22s
# kubectl get pod -o wide ##創(chuàng)建的pod
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
nginx-dbddb74b8-5smx7 1/1 Running 0 2m44s 172.17.59.2 10.40.6.213 <none>
nginx-dbddb74b8-hcjbw 1/1 Running 0 2m44s 172.17.31.2 10.40.6.210 <none>
nginx-dbddb74b8-jtwt5 1/1 Running 0 2m44s 172.17.59.3 10.40.6.213 <none>
###查看所有運(yùn)行的資源
# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/nginx-dbddb74b8-5smx7 1/1 Running 0 4m56s
pod/nginx-dbddb74b8-hcjbw 1/1 Running 0 4m56s
pod/nginx-dbddb74b8-jtwt5 1/1 Running 0 4m56s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 6h8m
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx 3 3 3 3 4m56s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-dbddb74b8 3 3 3 4m56s
管理層級(jí)：deployment -----> replicaset -----> pod
創(chuàng)建一個(gè)service，露88端口, 名稱為nginx , pod監(jiān)聽的80,
# kubectl expose deployment nginx --port=88 --target-port=80 --type=NodePort
# kubectl get svc ###查看Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 6h20m
nginx NodePort 10.0.0.153 <none> 88:40370/TCP 23s
## 88端口是對(duì)節(jié)點(diǎn)內(nèi)部，40370端口是對(duì)節(jié)點(diǎn)外部
訪問集群中部署的Nginx：
① node之間訪問地址：http://10.0.0.153:88， 如：curl http://10.0.0.153:88 -I
② node之外訪問地址：http://POD_IP:40370 ， 如：curl http://10.40.6.213:40370 -I
查看nginx pod日志權(quán)限問題：
# kubectl logs nginx-dbddb74b8-5smx7
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-5smx7)
說明system:anonymous 匿名用戶沒有獲取的權(quán)限，
要將這個(gè)用戶綁定到系統(tǒng)角色權(quán)限，使之有這角色權(quán)限。
集群角色的綁定：將一個(gè)用戶綁定到某個(gè)角色上，某個(gè)角色具備哪些權(quán)限
# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
# kubectl logs nginx-dbddb74b8-hcjbw -f
172.17.59.0 - - [30/May/2019:09:24:40 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
10.0.0.153 - - [30/May/2019:09:26:37 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.29.0" "-"
十. 部署Web UI（Dashboard）
UI YAML配置文件托管項(xiàng)目地址：
https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dashboard
dashboard-configmap.yaml 存放UI配置信息
dashboard-controller.yaml 控制器
dashboard-rbac.yaml 用于創(chuàng)建用戶并授權(quán)
dashboard-secret.yaml 存放敏感重要信息
dashboard-service.yaml 將UI暴露出來，讓我們?cè)L問
以上這些文件在之前下載的kubernetes-server-linux-amd64.tar.gz 包都存在，
# cd /usr/local/src/k8s/kubernetes && tar xvf kubernetes-src.tar.gz
# cd cluster ## 這個(gè)目錄是github 地址上的文件目錄一一對(duì)應(yīng)的，addons目錄下就是一些插件
# cd addons/dashboard/ ## 這就是存放以上的一些yaml配置文件
# ll
total 32
-rw-rw-r-- 1 root root 264 Oct 6 2018 dashboard-configmap.yaml
-rw-rw-r-- 1 root root 1821 Oct 6 2018 dashboard-controller.yaml
-rw-rw-r-- 1 root root 1353 Oct 6 2018 dashboard-rbac.yaml
-rw-rw-r-- 1 root root 551 Oct 6 2018 dashboard-secret.yaml
-rw-rw-r-- 1 root root 322 Oct 6 2018 dashboard-service.yaml
-rw-rw-r-- 1 root root 242 Oct 6 2018 MAINTAINERS.md
-rw-rw-r-- 1 root root 125 Oct 6 2018 OWNERS
-rw-rw-r-- 1 root root 400 Oct 6 2018 README.md
創(chuàng)建相應(yīng)的pod：
修改dashboard-controller.yaml配置文件鏡像地址
image: registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.10.1
瀏覽器打開地址：https://promotion.aliyun.com, 搜索kubernetes-dashboard-amd64這鏡像找到較新的鏡像
# kubectl create -f dashboard-configmap.yaml
# kubectl create -f dashboard-rbac.yaml
# kubectl create -f dashboard-secret.yaml
# kubectl create -f dashboard-controller.yaml
查看啟動(dòng)情況：
# kubectl get pod -n kube-system ## 命名空間為kube-system
NAME READY STATUS RESTARTS AGE
kubernetes-dashboard-774f47666c-97c86 1/1 Running 0 93s
# kubectl logs kubernetes-dashboard-774f47666c-97c86 -n kube-system
dashboard-service.yaml配置文件添加一個(gè)type: NodePort, 并創(chuàng)建一個(gè)service
# cat dashboard-service.yaml
apiVersion: v1
kind: Service
metadata:
name: kubernetes-dashboard
namespace: kube-system
labels:
k8s-app: kubernetes-dashboard
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
type: NodePort
selector:
k8s-app: kubernetes-dashboard
ports:
- port: 443
targetPort: 8443
# kubectl create -f dashboard-service.yaml
# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.0.0.198 <none> 443:30899/TCP 48s
瀏覽器訪問：https://10.40.6.210:30899
首次登錄會(huì)出現(xiàn)兩種驗(yàn)證方式，這里我們選擇token令牌驗(yàn)證方式：
選擇驗(yàn)證方式.png
要用令牌登錄，得先有個(gè)用戶身份，這個(gè)用戶可以用token標(biāo)志：
創(chuàng)建一個(gè)角色為ServiceAccount的dashboard-admin用戶,然后給用戶dashboard-admin綁定到cluster-admin角色，然后使用dashboard-admin產(chǎn)生的token 來登錄訪問，訪問apiserverpod也是使用rbc授權(quán)，pod 使用角色為ServiceAccount的dashboard-admin用戶訪問apiserver。
創(chuàng)建用戶及授權(quán)yaml配置文件如下：
# cat k8s-admin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: dashboard-admin
namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: dashboard-admin
subjects:
- kind: ServiceAccount
name: dashboard-admin
namespace: kube-system
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
創(chuàng)建token：
# kubectl create -f k8s-admin.yaml
# kubectl get secret -n kube-system
NAME TYPE DATA AGE
dashboard-admin-token-tbszw kubernetes.io/service-account-token 3 43s
查看token值：
# kubectl describe secret dashboard-admin-token-tbszw -n kube-system
Name: dashboard-admin-token-tbszw
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: dashboard-admin
kubernetes.io/service-account.uid: b962b8b9-82da-11e9-8a6c-005056b66bc1
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1359 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tdGJzenciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYjk2MmI4YjktODJkYS0xMWU5LThhNmMtMDA1MDU2YjY2YmMxIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.dNRBcXx-KdLr4tb2sqlKXchGdYPUxo2KoNnCH1ENae051P_7dE50SsJdN70eUR7pACo8LGbmPSVjnhIqYGTv4oS80bVBl1pZdYs1JS9Mc3jAG64npKLq_HfyMjsQSYW2c1Ial6WYRHIsqeegnVOy8vY22-gqSnPUYf1Sn5qYyJVRCy6yGMJ4P1Su1yBqRQO29rC-tgunEg28Rx339ADPoqsbKRCP3Q1Zwbkux1JBnXiGoZGKZjP_06lY3xAnmMzkI3wa4S5KQRIe68s6WH5RL-SWqkL5GiHWoz14CpkweiQ_4LUxH8zi_jQNH8Jsz3zd5eSYs2Pks5BKdj3-Drh17w
然后使用token值驗(yàn)證登錄UI
UI web.png
十一. 多Master集群-部署master01
多Master集群架構(gòu)圖.png
master 高可用主要是apiserver組件，scheduler調(diào)度器和controller-manager控制器本身就是高可用，可以通過配置文件可以知道，參數(shù)：
--leader-elect自動(dòng)選舉。apiserver 是以http方式提供對(duì)外服務(wù)，所以做http高可用方案可選擇nginx+keepalived或haproxy+keepalived等成熟方案，負(fù)載均衡器使用VIP，實(shí)現(xiàn)高可用。將master01節(jié)點(diǎn)的組件文件都拷貝到master02，并修改各個(gè)組件配置文件里的相應(yīng)IP即可：
# scp -r /opt/kubernetes 10.40.6.209:/opt/
# scp /usr/lib/systemd/system/{kube-apiserver,kube-scheduler,kube-controller-manager}.service 10.40.6.209:/usr/lib/systemd/system/
# scp -r /opt/etcd/ssl/ 10.40.6.209:/opt/etcd/ssl/
master02啟動(dòng)服務(wù)：
# systemctl start kube-apiserver
# systemctl start kube-scheduler
# systemctl start kube-controller-manager
master02查看集群狀態(tài)和節(jié)點(diǎn)：
# /opt/kubernetes/bin/kube
kube-apiserver kube-controller-manager kubectl kube-scheduler
[root@k8s logs]# /opt/kubernetes/bin/kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
# /opt/kubernetes/bin/kubectl get node
NAME STATUS ROLES AGE VERSION
10.40.6.210 Ready <none> 22h v1.12.1
10.40.6.213 Ready <none> 22h v1.12.1
node01和node02雖然沒有連接master02, 是因?yàn)閗ubernetes的集群狀態(tài)、配置、服務(wù)信息都存在etcd數(shù)據(jù)庫中，只要能連上etcd就能獲取集群相關(guān)信息。
十二. 多Master集群(Nginx+Keepalive)
這里使用nginx的4層負(fù)載均衡，分別在兩臺(tái)機(jī)器上安裝nginx：
# yum install yum-utils
# cat /etc/yum.repos.d/nginx.repo
[nginx-stable] ###穩(wěn)定庫，默認(rèn)使用穩(wěn)定庫
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
[nginx-mainline] ###主線庫
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key
# yum install nginx
配置文件：/etc/nginx/nginx.conf
stream字段與http字段同級(jí)，配置文件添加如下配置：
stream {
log_format main "$remote_addr $upstream_addr $time_local $status";
access_log /var/log/nginx/k8s_apiserver-accese.log main;
upstream k8s-apiserver {
server 10.40.6.201:6443;
server 10.40.6.209:6443;
}
server {
listen 6443;
proxy_pass k8s-apiserver;
}
}
啟動(dòng)報(bào)錯(cuò)：bind() to 0.0.0.0:6443 failed (13: Permission denied)
原因：監(jiān)聽端口6443 不在http允許訪問的端口
操作：
# semanage port -l | grep http_port_t
http_port_t tcp 80, 81, 443, 488, 8008, 8009, 8443, 9000
pegasus_http_port_t tcp 5988
# semanage port -a -t http_port_t -p tcp 6443
# systemctl start nginx
修改node兩個(gè)節(jié)點(diǎn)三個(gè)配置文件中連接apiserver的地址為nginx IP地址：
# grep 6443 ./*
./bootstrap.kubeconfig: server: https://10.40.6.166:6443
./kubelet.kubeconfig: server: https://10.40.6.166:6443
./kube-proxy.kubeconfig: server: https://10.40.6.166:6443
重啟kubelet：
# systemctl restart kubelet
# systemctl restart kube-proxy
在任何一臺(tái)master查看集群狀態(tài)：
# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
[root@k8s ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
10.40.6.210 Ready <none> 23h v1.12.1
10.40.6.213 Ready <none> 23h v1.12.1
可以通過查看nginx 請(qǐng)求日志驗(yàn)證是否請(qǐng)求正常。
接著在nginx兩臺(tái)機(jī)器安裝keepalived：
# yum install keepalived -y
配置文件：
# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51 # VRRP 路由 ID實(shí)例，每個(gè)實(shí)例是唯一的
priority 90 # 優(yōu)先級(jí)，備服務(wù)器設(shè)置 90
advert_int 1 # 指定VRRP 心跳包通告間隔時(shí)間，默認(rèn)1秒
authentication {
auth_type PASS
auth_pass jNikdfK8
}
virtual_ipaddress {
10.40.6.175/23
}
track_script {
check_nginx
}
}
nginx狀態(tài)檢測腳本：
# cat /etc/keepalived/check_nginx.sh
#!/bin/bash
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
systemctl stop keepalived.service
fi
Nginx+Keepalived配置測試VIP漂移沒問題后，將兩個(gè)node節(jié)點(diǎn)請(qǐng)求apiserver IP 改為VIP的IP 10.40.6.175
# grep 6443 /opt/kubernetes/cfg/*
/opt/kubernetes/cfg/bootstrap.kubeconfig: server: https://10.40.6.175:6443
/opt/kubernetes/cfg/kubelet.kubeconfig: server: https://10.40.6.175:6443
/opt/kubernetes/cfg/kube-proxy.kubeconfig: server: https://10.40.6.175:6443
# systemctl restart kubelet
# systemctl restart kube-proxy
然后重啟某個(gè)node的kubelet，查看nginx日志文件，可以發(fā)現(xiàn)請(qǐng)求OK
