環(huán)境準(zhǔn)備

練習(xí)環(huán)境采用VirtualBox運(yùn)行ubuntu18.04，有現(xiàn)成的機(jī)器此環(huán)節(jié)可略過(guò)。

1、VirtualBox設(shè)置

1、新建虛擬電腦
2、設(shè)置》系統(tǒng)》cpu 2，內(nèi)存2G（k8s最低要求）
3、下載Ubuntu Server 18.04 LTS，設(shè)置》存儲(chǔ)》沒(méi)有盤(pán)片》分配光驅(qū)，
選擇剛剛下載的ubuntu鏡像文件
4、啟動(dòng)，初次注意勾選安裝sshd
5、配置網(wǎng)絡(luò)：在VertualBox中，將虛擬機(jī)網(wǎng)絡(luò)設(shè)置為橋接模式；參考宿主機(jī)網(wǎng)絡(luò)信息，
修改虛擬機(jī)文件cat /etc/netplan/50-cloud-init.yaml，然后 netplan apply

network:
    ethernets:
        enp0s3:
            addresses: [192.168.0.191/24]            # [ip/子網(wǎng)掩碼]
            gateway4: 192.168.0.1                    # 網(wǎng)關(guān)
            dhcp4: no                               # 是否動(dòng)態(tài)獲取ip
            nameservers:
                    addresses: [114.114.114.114]    # dns
    version: 2

備注：子網(wǎng)掩碼設(shè)置請(qǐng)搜索CIDR（無(wú)類別域間路由，Classless Inter-Domain Routing），點(diǎn)擊查看可用dns
6、安裝docker apt-get install -y docker.io

2、安裝k8s（阿里云源）

• 添加軟件包密鑰(apt-key)
  sudo curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg --output /etc/apt/trusted.gpg.d/apt-key.gpg
• 添加源

# /etc/apt/sources.list.d/kubernetes.list
deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main

• 安裝
  sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl

3、創(chuàng)建集群

1、確認(rèn)機(jī)器開(kāi)放端口， 如果使用aws或者阿里云機(jī)器務(wù)必配置開(kāi)啟相應(yīng)端口的安全組

master節(jié)點(diǎn)開(kāi)放端口

node 節(jié)點(diǎn)開(kāi)放端口

2、所有節(jié)點(diǎn)啟用 docker service，關(guān)閉 swap
systemctl enable docker.service
swapoff -a
3、在master啟動(dòng)kubeadm
sudo kubeadm init --pod-network-cidr=182.168.0.0/16 --image-repository registry.aliyuncs.com/google_containers
輸出：

Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.191:6443 --token lgm81w.xjktiv2w32jozvgt --discovery-token-ca-cert-hash sha256:a3a535428d2302acfc4cdff1dc3a6db29bd01bbbbccae824f902842523035edd

4、 設(shè)置 pod network calico
下載網(wǎng)絡(luò)插件配置文件 calico.yaml
curl https://docs.projectcalico.org/v3.6/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml > calico.yaml
注意修改參數(shù) CALICO_IPV4POOL_CIDR，與 kubeadm init 的參數(shù) pod-network-cidr保持一致！

# The default IPv4 pool to create on startup if none exists. Pod IPs will be
# chosen from this range. Changing this value after installation will have
# no effect. This should fall within `--cluster-cidr`.
- name: CALICO_IPV4POOL_CIDR
  value: "182.168.0.0/16"

執(zhí)行 sudo kubectl apply -f calico.yaml

5、加入節(jié)點(diǎn)，在節(jié)點(diǎn)機(jī)器執(zhí)行命令

yetongxue@vm2:~$ kubeadm join 192.168.0.191:6443 --token lgm81w.xjktiv2w32jozvgt --discovery-token-ca-cert-hash sha256:a3a535428d2302acfc4cdff1dc3a6db29bd01bbbbccae824f902842523035edd
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

在 master 檢查 vm2 加入狀態(tài)

yetongxue@vm1:~$ sudo kubectl get nodes
NAME   STATUS   ROLES    AGE   VERSION
vm1    Ready    master   21m   v1.15.0
vm2    Ready    <none>   19m   v1.15.0

如果出現(xiàn)問(wèn)題，執(zhí)行kubectl describe nodes <node_name>查看原因
k8s環(huán)境安裝完成 ??

部署應(yīng)用集群

前置知識(shí)

• pod 一個(gè)或一組容器組成，共同提供一種服務(wù)；定義文件pod.yaml中包含使用的鏡像、端口、數(shù)據(jù)卷掛載、啟動(dòng)命令，為pod打標(biāo)簽label這些，類似編排多個(gè)容器的 docker-compose.yaml 的作用
• service 多個(gè)相同服務(wù)的pod組成一個(gè)服務(wù)，通過(guò)service.yaml中spec.selecter=[pod label]選擇pod，負(fù)載均衡的感覺(jué)
• volume 數(shù)據(jù)卷，有很多類型：emptyDir適合pod內(nèi)部各個(gè)容器共享內(nèi)容，hostPath則會(huì)將目錄掛載到宿主機(jī)，還有網(wǎng)絡(luò)數(shù)據(jù)卷、分布式文件系統(tǒng)這些；還有一種專門當(dāng)做配置文件來(lái)用的數(shù)據(jù)掛載方式——configMap，可以用一份文件來(lái)生成configMap，用來(lái)替換某些軟件原來(lái)的配置文件；也可以設(shè)置很多的key value，在各種yaml文件中引用，以設(shè)置容器所需要的環(huán)境變量參數(shù)

參考資料

• kubernetes yaml格式的pod定義文件完整內(nèi)容
• Creating the pod using the YAML file
• config map
• 通過(guò)configmap更新k8s里的mysql配置文件
• kubernetes 特殊存儲(chǔ)卷ConfigMap

下面就將項(xiàng)目 django_demo 部署到 k8s 集群，參考 docker-compose.yaml、mysqld.cnf內(nèi)容，分別創(chuàng)建如下文件（k8s命名一般是小駝峰/中劃線）

1、通過(guò)文件 mysqld.cnf 創(chuàng)建 configmap

yetongxue@vm1:~$ sudo kubectl create configmap mysqld-config --from-file=mysqld.cnf
configmap/mysqld-config created
yetongxue@vm1:~$ sudo kubectl describe configmaps  mysqld-config
[sudo] password for master: 
Name:         mysqld-config
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
mysqld.cnf:
----
[mysqld]
pid-file    = /var/run/mysqld/mysqld.pid
socket        = /var/run/mysqld/mysqld.sock
datadir       = /var/lib/mysql
#log-error  = /var/log/mysql/error.log
# By default we only accept connections from localhost
bind-address  = 0.0.0.0
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

# utf8mb4
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
init_connect='SET NAMES utf8mb4'
skip-character-set-client-handshake = true

Events:  <none>

2、將 configmap mysql-config 掛載到 pod，替換 mysql 鏡像原有的myqld.cnf；將mysql數(shù)據(jù)目錄（見(jiàn)mysqd.cnf 中 datadir 字段）/var/lib/mysql 掛載到宿主機(jī)/var/lib/mysql-datadir

# mysql-pod.yaml
apiVersion: v1
kind: Pod
metadata:
    name: mysql
    labels:
        app: database
        version: v1
        release: dev
spec:
    restartPolicy: Always
    containers:
        - name: mysql
          image: mysql:5.7
          imagePullPolicy: IfNotPresent
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: MYSQL_ROOT_PASSWORD
              value: root
            - name: MYSQL_USER
              value: yetongxue
            - name: MYSQL_PASSWORD
              value: qwerasdf
            - name: MYSQL_DATABASE
              value: django_demo
          volumeMounts:
            - name: config-volume-mysql
              mountPath: /etc/mysql/mysql.conf.d/
            - name: mysql-datadir
              mountPath: /var/lib/mysql
    volumes:
        - name: config-volume-mysql
          configMap:
            name: mysqld-config
        - name: mysql-datadir
          hostPath:
            path: /var/lib/mysql-datadir

啟動(dòng)并驗(yàn)證 mysql pod 是否正常

yetongxue@vm1:~$ sudo kubectl apply -f mysql-pod.yaml 
pod/mysql created
yetongxue@vm1:~$ sudo kubectl get pod mysql
NAME    READY   STATUS    RESTARTS   AGE
mysql   1/1     Running   0          9d

yetongxue@vm1:~$ sudo kubectl exec -it mysql bash
root@mysql:/# mysql -u root -p
Enter password: 
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| django_demo        |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

檢查數(shù)據(jù)掛載是否成功

yetongxue@vm2:~$ ls /var/lib/mysql-datadir/
auto.cnf    ca.pem           client-key.pem  ib_buffer_pool  ib_logfile0  ibtmp1  performance_schema  public_key.pem   server-key.pem
ca-key.pem  client-cert.pem  django_demo     ibdata1         ib_logfile1  mysql   private_key.pem     server-cert.pem  sys

這里有一個(gè)優(yōu)化的地方，因?yàn)?mysql-pod.yaml 中使用了許多的環(huán)境變量，這些變量在下面的 django-demo 部署中依然會(huì)使用到，所以我們將這些變量剝離出來(lái)，通過(guò) configMap 引用。
創(chuàng)建 mysql 環(huán)境變量 configMap，參考 Configure all key-value pairs in a ConfigMap as container environment variables

# env-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: env-config
data:
  MYSQL_ROOT_PASSWORD: root
  MYSQL_USER: yetongxue
  MYSQL_PASSWORD: qwerasdf
  MYSQL_DATABASE: django_demo

使用前先創(chuàng)建 configMap

yetongxue@vm1:~$ sudo kubectl apply -f env-config.yaml 
configmap/env-config created

下面將 mysql-pod.yaml 改造一下，環(huán)境變量通過(guò) configMap 的方式引用，如下：

# mysql-pod.yaml
apiVersion: v1
kind: Pod
metadata:
    name: mysql
    labels:
        app: database
        version: v1
        release: dev
spec:
    restartPolicy: Always
    containers:
        - name: mysql
          image: mysql:5.7
          imagePullPolicy: IfNotPresent
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: env-config
                  key: MYSQL_ROOT_PASSWORD
            - name: MYSQL_USER
              valueFrom:
                configMapKeyRef:
                  name: env-config
                  key: MYSQL_USER
            - name: MYSQL_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: env-config
                  key: MYSQL_PASSWORD 
            - name: MYSQL_DATABASE
              valueFrom:
                configMapKeyRef:
                  name: env-config
                  key: MYSQL_DATABASE 
          volumeMounts:
            - name: config-volume-mysql
              mountPath: /etc/mysql/mysql.conf.d/
            - name: mysql-datadir
              mountPath: /var/lib/mysql
    volumes:
        - name: config-volume-mysql
          configMap:
            name: mysqld-config
        - name: mysql-datadir
          hostPath: 
            path: /var/lib/mysql-datadir 

這里還有一個(gè)優(yōu)化的地方 ??，一般我們都不通過(guò) pod.yaml 直接創(chuàng)建pod，而是通過(guò) Deployment Controller 來(lái)自動(dòng)創(chuàng)建 pod，Deployment 定義文件 deployment.yaml 中參數(shù)replicas設(shè)置期望運(yùn)行的 pod 數(shù)量，Deployment 會(huì)自動(dòng)增加或者刪除 pod 來(lái)維持；另外參數(shù)template的內(nèi)容便是剛剛的 pod.yaml 內(nèi)容，所以 RC 便知道如何去創(chuàng)建 pod；彈性伸縮、滾動(dòng)升級(jí)也是通過(guò)它來(lái)實(shí)現(xiàn)的。

# mysql-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
    name: mysql-deployment
spec:
    selector:
        matchLabels:
            app: database
    replicas: 1
    template:
        metadata:
            name: mysql
            labels:
                app: database
        spec:
            restartPolicy: Always
            containers:
            - name: mysql
              image: mysql:5.7
              imagePullPolicy: IfNotPresent
              env:
                - name: TZ
                  value: Asia/Shanghai
                - name: MYSQL_ROOT_PASSWORD
                  valueFrom:
                    configMapKeyRef:
                      name: env-config
                      key: MYSQL_ROOT_PASSWORD
                - name: MYSQL_USER
                  valueFrom:
                    configMapKeyRef:
                      name: env-config
                      key: MYSQL_USER
                - name: MYSQL_PASSWORD
                  valueFrom:
                    configMapKeyRef:
                      name: env-config
                      key: MYSQL_PASSWORD 
                - name: MYSQL_DATABASE
                  valueFrom:
                    configMapKeyRef:
                      name: env-config
                      key: MYSQL_DATABASE 
              volumeMounts:
                - name: config-volume-mysql
                  mountPath: /etc/mysql/mysql.conf.d/
                - name: mysql-datadir
                  mountPath: /var/lib/mysql
            volumes:
              - name: config-volume-mysql
                configMap:
                    name: mysqld-config
              - name: mysql-datadir
                hostPath: 
                    path: /var/lib/mysql-datadir

刪除剛剛創(chuàng)建的 pod: sudo kubectl delete pod mysql
通過(guò) RC 創(chuàng)建 pod: sudo kubectl apply -f mysql-deployment.yaml

這里有兩個(gè)地方需要注意：

• spec.selector.matchLabels務(wù)必和 spec.template.matadata.labels保持一致，否則RC通過(guò)template創(chuàng)建出pod后，通過(guò)matchLabels匹配不到相應(yīng)的pod就會(huì)不停的創(chuàng)建。
• 對(duì)于mysql這種有數(shù)據(jù)狀態(tài)的應(yīng)用，不能簡(jiǎn)單的設(shè)置replicas為大于1的數(shù)值來(lái)擴(kuò)容，這又是個(gè)優(yōu)化的地方，有興趣的同學(xué)可參考 Run a Replicated Stateful Application

3、創(chuàng)建 msyql service

Kubernetes 中 service 有三種類型:

• ClusterIP:提供一個(gè)集群內(nèi)部的虛擬IP以供Pod訪問(wèn)。
• NodePort:在每個(gè)Node上打開(kāi)一個(gè)端口以供外部訪問(wèn)。
• LoadBalancer:通過(guò)外部的負(fù)載均衡器來(lái)訪問(wèn)。

顯然，這里的mysql用ClusterIP就好；待會(huì)下面創(chuàng)建django-demo service就用LoadBalancer。

# mysql-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: mysql-service        
spec:
  selector:
    app: database    #通過(guò)label選擇加入service的pod
  ports:                    
  - port: 3306
    targetPort: 3306
    protocol: TCP

啟動(dòng)mysql service

yetongxue@vm1:~$ sudo kubectl apply -f mysql-service.yaml 
service/mysql-service created
yetongxue@vm1:~$ sudo kubectl get service
NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
kubernetes      ClusterIP   10.96.0.1        <none>        443/TCP    10h
mysql-service   ClusterIP   10.108.192.143   <none>        3306/TCP   46s
yetongxue@vm1:~$ sudo kubectl get endpoints mysql-service 
NAME            ENDPOINTS              AGE
mysql-service   182.168.185.194:3306   24m

ENDPOINTS表明已經(jīng)將剛剛創(chuàng)建的mysql pod納入了mysql service.

4、創(chuàng)建 django-demo deployment

先把通過(guò) Dockerfile 構(gòu)建的鏡像上傳到阿里云鏡像倉(cāng)庫(kù) registry.cn-hangzhou.aliyuncs.com/yetongxue/django_demo:dev。
由于RC的好處是明顯的，所以現(xiàn)在我們就直接通過(guò)RC來(lái)創(chuàng)建pod
注意將MYSQL_HOST修改成剛剛創(chuàng)建的mysql-service的ClusterIP，或者直接就用service name: mysql-service

# django-demo-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
    name: django-demo-deployment
spec:
    selector:
        matchLabels:
            app: web
    replicas: 2
    template:
        metadata:
            name: django-demo
            labels:
                app: web
        spec:
            restartPolicy: Always
            containers:
              - name: demo
                image: registry.cn-hangzhou.aliyuncs.com/yetongxue/django_demo:dev
                imagePullPolicy: IfNotPresent
                env:
                  - name: TZ
                    value: Asia/Shanghai
                  - name: MYSQL_HOST
                    value: mysql-service
                  - name: MYSQL_USER
                    valueFrom:
                      configMapKeyRef:
                        name: env-config
                        key: MYSQL_USER
                  - name: MYSQL_PASSWORD
                    valueFrom:
                      configMapKeyRef:
                        name: env-config
                        key: MYSQL_PASSWORD 
                  - name: MYSQL_DATABASE
                    valueFrom:
                      configMapKeyRef:
                        name: env-config
                        key: MYSQL_DATABASE
                ports:
                  - containerPort: 80

啟動(dòng) django-demo deployment，并驗(yàn)證連接數(shù)據(jù)庫(kù)是否正常。

yetongxue@vm1:~$ sudo kubectl apply -f django-demo-deployment.yaml 
deployment.apps/django-demo-deployment created
yetongxue@vm1:~$ sudo kubectl get pods
NAME                                      READY   STATUS              RESTARTS   AGE
django-demo-deployment-8477cd6dbb-4dz9b   0/1     ContainerCreating   0          57s
django-demo-deployment-8477cd6dbb-7jssg   0/1     ContainerCreating   0          57s
mysql-deployment-588c96799f-7md5n         1/1     Running             0          24m
yetongxue@vm1:~$ sudo kubectl exec -it mysql-deployment-588c96799f-7md5n bash
root@mysql-deployment-588c96799f-7md5n:/# mysql -u root -p
Enter password: 
mysql> use django_demo
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+--------------------------------+
| Tables_in_django_demo          |
+--------------------------------+
| auth_group                     |
| auth_group_permissions         |
| auth_permission                |
| authtoken_token                |
| django_admin_log               |
| django_content_type            |
| django_migrations              |
| django_session                 |
| test_app_book                  |
| test_app_user                  |
| test_app_user_groups           |
| test_app_user_user_permissions |
+--------------------------------+
12 rows in set (0.01 sec)

5、創(chuàng)建django-demo service

# django-demo-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: django-demo-service
spec:
  selector:
    app: web
  externalIPs: ["192.168.0.191"]      # 指定一個(gè)外部IP，這里就用master節(jié)點(diǎn)IP
  ports:
  - name: http
    port: 8000
    targetPort: 80
    protocol: TCP
  type: LoadBalancer        # 類型負(fù)載均衡

啟動(dòng)并驗(yàn)證是否成功

yetongxue@vm1:~$ sudo kubectl apply -f django-demo-service.yaml 
service/django-demo-service created

訪問(wèn)地址：http://192.168.0.191:8000/admin
賬號(hào)：admin
密碼：qwerasdf
截圖紀(jì)念如下：

image.png

問(wèn)題收集

在參照文檔進(jìn)行練習(xí)時(shí)，遇到的一些問(wèn)題記錄如下

1、在master執(zhí)行kubeadm reset之后，如果你之前執(zhí)行過(guò)命令sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config，則需要再次執(zhí)行，否則權(quán)限無(wú)法驗(yàn)證，因?yàn)?HOME/.kube/config里面存的還是上次kubeadm init生成的文件；node被移除或者master執(zhí)行了kubeadm reset， 在執(zhí)行kubeadm join之前還需要執(zhí)行 kubeadm reset，否則在進(jìn)行 kubeadm join時(shí)產(chǎn)生如下錯(cuò)誤

[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[kubelet-check] Initial timeout of 40s passed.
packet_write_wait: Connection to 192.168.0.192 port 22: Broken pipe

2、在使用VirtualBox時(shí)，安裝完ubuntu我reboot了（以后一定要通過(guò)VirtualBox操作），結(jié)果當(dāng)前虛擬機(jī)在VirtualBox的狀態(tài)就變成了Guru Meditation，無(wú)法關(guān)閉啟動(dòng)刪除，只可查看日志，又沒(méi)看懂日志。不知道為啥，最后直接kill相應(yīng)進(jìn)程解決

常用命令

• 查看 pod: kubectl describe pod --namespace=<name_space> <pod_name>
• 進(jìn)入pod: kubectl exec -it <pod_name> bash
• 查看pod日志：kubectl logs <pod_name>