一、前言
上一文《從零開始搭建Kubernetes集群（四、搭建K8S Dashboard）》介紹了如何搭建Dashboard。本篇將介紹如何搭建Ingress來訪問K8S集群的Service。
二、Ingress簡介
Ingress是個什么鬼，網上資料很多（推薦官方），大家自行研究。簡單來講，就是一個負載均衡的玩意，其主要用來解決使用NodePort暴露Service的端口時Node IP會漂移的問題。同時，若大量使用NodePort暴露主機端口，管理會非常混亂。
好的解決方案就是讓外界通過域名去訪問Service，而無需關心其Node IP及Port。那為什么不直接使用Nginx？這是因為在K8S集群中，如果每加入一個服務，我們都在Nginx中添加一個配置，其實是一個重復性的體力活，只要是重復性的體力活，我們都應該通過技術將它干掉。
Ingress就可以解決上面的問題，其包含兩個組件Ingress Controller和Ingress：
- Ingress
將Nginx的配置抽象成一個Ingress對象，每添加一個新的服務只需寫一個新的Ingress的yaml文件即可 - Ingress Controller
將新加入的Ingress轉化成Nginx的配置文件并使之生效
好了，廢話不多，走你~
三、準備操作
官方文檔
人生苦短，不造輪子，本文將以官方的標準腳本為基礎進行搭建，參考請戳官方文檔。官方文檔中要求依次執行如下命令：
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/udp-services-configmap.yaml \
| kubectl apply -f -
以上yaml文件創建Ingress用到的Namespace、ConfigMap，以及默認的后端default-backend。最關鍵的一點是，由于之前我們基于Kubeadm創建了K8S集群，則還必須執行：
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/rbac.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml \
| kubectl apply -f -
這是由于Kubeadm創建的集群默認開啟了RABC，因此Ingress也必須創建相應的RABC權限控制。
導入鏡像
但是，直接按照上述方式執行，我們的Ingress很可能會無法使用。所以，我們需要將上述Yaml文件全部wget下來，經過一些修改后才能執行kubectl apply -f創建。另外需要注意的是，這些yaml文件中提到的一些鏡像，國內目前無法下載，如：
gcr.io/google_containers/defaultbackend:1.4
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
本人已經提前下載好，大家請戳：
地址：https://pan.baidu.com/s/1N-bK9hI7JTZZB6AzmaT8PA
密碼：1a8a
拿到鏡像后，在每個節點上執行如下命令導入鏡像：
docker load < quay.io#kubernetes-ingress-controller#nginx-ingress-controller_0.14.0.tar
docker tag 452a96d81c30 quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
docker load < gcr.io#google_containers#defaultbackend.tar
docker tag 452a96d81c30 gcr.io/google_containers/defaultbackend
如上所示，導入鏡像后，別忘記給打tag，否則鏡像名稱為<none>：
image.png
四、主要文件介紹
這里，我們先對一些重要的文件進行簡單介紹。
default-backend.yaml
default-backend的作用是，如果外界訪問的域名不存在的話，則默認轉發到default-http-backend這個Service，其會直接返回404：
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: default-http-backend
labels:
app: default-http-backend
namespace: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app: default-http-backend
template:
metadata:
labels:
app: default-http-backend
spec:
terminationGracePeriodSeconds: 60
containers:
- name: default-http-backend
# Any image is permissible as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: gcr.io/google_containers/defaultbackend:1.4
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
namespace: ingress-nginx
labels:
app: default-http-backend
spec:
ports:
- port: 80
targetPort: 8080
selector:
app: default-http-backend
rbac.yaml
rbac.yaml負責Ingress的RBAC授權的控制，其創建了Ingress用到的ServiceAccount、ClusterRole、Role、RoleBinding、ClusterRoleBinding。在上文《從零開始搭建Kubernetes集群（四、搭建K8S Dashboard）》中，我們已對這些概念進行了簡單介紹。
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
with-rbac.yaml
with-rbac.yaml是Ingress的核心，用于創建ingress-controller。前面提到過，ingress-controller的作用是將新加入的Ingress進行轉化為Nginx的配置。
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app: ingress-nginx
template:
metadata:
labels:
app: ingress-nginx
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --annotations-prefix=nginx.ingress.kubernetes.io
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
securityContext:
runAsNonRoot: false
如上，可以看到nginx-ingress-controller啟動時傳入了參數，分別為前面創建的default-backend-service以及configmap。
五、創建Ingress
1.創建Ingress-controller
需要注意的是，官方提供的with-rbac.yaml文件不能直接使用，我們必須修改兩處：
加入hostNetwork配置
如下，在serviceAccountName上方添加hostNetwork: true:
spec:
hostNetwork: true
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --annotations-prefix=nginx.ingress.kubernetes.io
配置hostNetwork: true是一種直接定義Pod網絡的方式。定義后，Ingress-controller的IP就與宿主機k8s-node1一樣(192.168.56.101)，并且端口80也是宿主機上的端口。這樣，我們通過該192.168.56.101:80，就可以直接訪問到Ingress-controller（實際上就是nginx），然后Ingress-controller則會轉發我們的請求到相應后端。
加入環境變量
在其env部分加入如下環境變量：
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: KUBERNETES_MASTER
value: http://192.168.56.101:8080
否則，創建后會提示如下錯誤：
[root@k8s-node1 ingress]# kubectl describe pod nginx-ingress-controller-9fbd7596d-rt9sf -n ingress-nginx
省略前面...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 30s default-scheduler Successfully assigned nginx-ingress-controller-9fbd7596d-rt9sf to k8s-node1
Normal SuccessfulMountVolume 30s kubelet, k8s-node1 MountVolume.SetUp succeeded for volume "nginx-ingress-serviceaccount-token-lq2dt"
Warning BackOff 21s kubelet, k8s-node1 Back-off restarting failed container
Normal Pulled 11s (x3 over 29s) kubelet, k8s-node1 Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0" already present on machine
Normal Created 11s (x3 over 29s) kubelet, k8s-node1 Created container
Warning Failed 10s (x3 over 28s) kubelet, k8s-node1 Error: failed to start container "nginx-ingress-controller": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/nginx-ingress-controller\": stat /nginx-ingress-controller: no such file or directory": unknown
修改with-rbac.yaml后，使用kubectl -f create命令分別執行如下yaml文件，即可創建Ingress-controller：
image.png
創建成功后如下所示：
[root@k8s-node1 ingress]# kubectl get pod -n ingress-nginx -o wide
NAME READY STATUS RESTARTS AGE IP NODE
default-http-backend-5c6d95c48-pdjn9 1/1 Running 0 23s 192.168.36.81 k8s-node1
nginx-ingress-controller-547cd7d9cb-jmvpn 1/1 Running 0 8s 192.168.36.82 k8s-node1
2.創建自定義Ingress
有了ingress-controller，我們就可以創建自定義的Ingress了。這里已提前搭建好了Kibana服務，我們針對Kibana創建一個Ingress：
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kibana-ingress
namespace: default
spec:
rules:
- host: myk8s.com
http:
paths:
- path: /
backend:
serviceName: kibana
servicePort: 5601
其中：
- rules中的host必須為域名，不能為IP，表示Ingress-controller的Pod所在主機域名，也就是Ingress-controller的IP對應的域名。
- paths中的path則表示映射的路徑。如映射
/表示若訪問myk8s.com，則會將請求轉發至Kibana的service，端口為5601。
創建成功后，查看：
[root@k8s-node1 ingress]# kubectl get ingress -o wide
NAME HOSTS ADDRESS PORTS AGE
kibana-ingress myk8s.com 80 6s
我們再執行kubectl exec nginx-ingress-controller-5b79cbb5c6-2zr7f -it cat /etc/nginx/nginx.conf -n ingress-nginx，可以看到生成nginx配置，篇幅較長，各位自行篩選：
## start server myk8s.com
server {
server_name myk8s.com ;
listen 80;
listen [::]:80;
set $proxy_upstream_name "-";
location /kibana {
log_by_lua_block {
}
port_in_redirect off;
set $proxy_upstream_name "";
set $namespace "kube-system";
set $ingress_name "dashboard-ingress";
set $service_name "kibana";
client_max_body_size "1m";
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering "off";
proxy_buffer_size "4k";
proxy_buffers 4 "4k";
proxy_request_buffering "on";
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
proxy_next_upstream_tries 0;
# No endpoints available for the request
return 503;
}
location / {
log_by_lua_block {
}
port_in_redirect off;
set $proxy_upstream_name "";
set $namespace "default";
set $ingress_name "kibana-ingress";
set $service_name "kibana";
client_max_body_size "1m";
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering "off";
proxy_buffer_size "4k";
proxy_buffers 4 "4k";
proxy_request_buffering "on";
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
proxy_next_upstream_tries 0;
# No endpoints available for the request
return 503;
}
}
## end server myk8s.com
3.設置host
首先，我們需要在Ingress-controller的Pod所在主機上（這里為k8s-node1），將上面提到的域名myk8s.com追加入/etc/hosts文件：
192.168.56.101 myk8s.com
除此之外，如果想在自己的Windows物理機上使用瀏覽器訪問kibana，也需要在C:\Windows\System32\drivers\etc\hosts文件內加入上述內容。設置后，分別在k8s-node1和物理機上測試無誤即可：
image.png
image.png
六、測試
在Windows物理機上，使用Chrome訪問myk8s.com，也就是相當于訪問了192.168.56.101:80：
image.png
隨意訪問一個錯誤的地址myk8s.com/abc，返回預期的404：
image.png
七、廢話
至此，我們的Ingress已經搭建完畢，實現了在外部通過域名訪問K8S集群Service的功能。如果大家有興趣，可以嘗試為Ingress配置TLS，這樣就可以訪問如Dashboard這種https服務了。下一章節《從零開始搭建Kubernetes集群（五、在K8S上部署Redis 集群）》，敬請期待。
本人水平有限，難免有錯誤或遺漏之處，望大家指正和諒解，歡迎評論留言。
