背景

本文主要講解kubernetes 如何使用helm chart安裝 ingress-nginx

kubernetes 使用helm chart安裝 ingress-nginx

1. ingres nginx controller 和 k8s版本 兼容性要求

https://github.com/kubernetes/ingress-nginx/blob/main/README.md#supported-versions-table
Supported Ingress-NGINX version k8s supported version Alpine Version Nginx Version Helm Chart Version
?? v1.11.2 1.30, 1.29, 1.28, 1.27, 1.26 3.20.0 1.25.5 4.11.2
?? v1.11.1 1.30, 1.29, 1.28, 1.27, 1.26 3.20.0 1.25.5 4.11.1
?? v1.11.0 1.30, 1.29, 1.28, 1.27, 1.26 3.20.0 1.25.5 4.11.0
?? v1.10.4 1.30, 1.29, 1.28, 1.27, 1.26 3.20.0 1.25.5 4.10.4
?? v1.10.3 1.30, 1.29, 1.28, 1.27, 1.26 3.20.0 1.25.5 4.10.3

2.安裝環(huán)境

注: linux amd64 替換相關(guān)鏡像版本 & helm3安裝包即可-安裝流程同理

linux arm64: 內(nèi)核版本 4.18.0-348.20.1.el7.aarch64 #1 SMP Wed Apr 13 20:57:50 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
Kubernetes: v1.28.0
Docker: 26.1.4
ingress-nginx: 4.11.2

3.安裝離線鏡像準(zhǔn)備

# 下載 ingress-controller依賴鏡像(國內(nèi)機(jī)器有墻無法拉取)
docker pull registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3
docker pull registry.k8s.io/ingress-nginx/controller:v1.11.2
# 導(dǎo)出為離線鏡像
docker save -o kube-webhook-certgen-v1.4.3.tar registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3
docker save -o controller-v1.11.2.tar registry.k8s.io/ingress-nginx/controller:v1.11.2
# 安裝k8s所有機(jī)器節(jié)點(diǎn)導(dǎo)入離線鏡像
docker load -i controller-v1.11.2.tar
docker load -i kube-webhook-certgen-v1.4.3.tar

# docker images|grep ingress
registry.k8s.io/ingress-nginx/controller             v1.11.2   289a818c8d9c   2 weeks ago     294MB
registry.k8s.io/ingress-nginx/kube-webhook-certgen   v1.4.3    420193b27261   3 weeks ago     53.3MB


# 鏡像打tag & push到本地倉庫[可選]
#docker tag registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3 sealos.hub:5000/ingress-nginx/kube-webhook-certgen:v1.4.3
#docker push sealos.hub:5000/ingress-nginx/kube-webhook-certgen:v1.4.3
#docker tag registry.k8s.io/ingress-nginx/controller:v1.11.2 sealos.hub:5000/ingress-nginx/controller:v1.11.2
#docker push sealos.hub:5000/ingress-nginx/controller:v1.11.2

4.linux(amd64)安裝 helm3

參考 https://helm.sh/zh/docs/intro/install/
https://github.com/helm/helm/releases

wget https://get.helm.sh/helm-v3.15.4-linux-amd64.tar.gz
tar -xvf helm-v3.15.4-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm
helm version

5.創(chuàng)建k8s拉取鏡像-鏡像倉庫驗(yàn)證鑒權(quán)信息

# 創(chuàng)建鏡像倉庫驗(yàn)證鑒權(quán)信息: k8s拉取驗(yàn)證 結(jié)合 imagePullSecrets: imagePullSecrets: - name: scr 引用使用
kubectl create secret docker-registry scr \
  -n ingress-nginx \
  --docker-server=http://sealos.hub:5000 \
  --docker-username=admin \
  --docker-password=123456 \
  --docker-email=jinze@ali.com
# 刪除鏡像倉庫驗(yàn)證鑒權(quán)信息
kubectl delete secret -n ingress-nginx scr

# 查看解密secret內(nèi)容
kubectl get secret  -n ingress-nginx scr --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode  

4.解壓安裝ingress-nginx

# helmchart 安裝 ingrss nginx
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

# helm列出所有版本:
helm search repo ingress-nginx/ingress-nginx -l
NAME                            CHART VERSION   APP VERSION     DESCRIPTION                                       
ingress-nginx/ingress-nginx     4.11.2          1.11.2          Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx     4.11.1          1.11.1          Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx     4.11.0          1.11.0          Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx     4.10.4          1.10.4          Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx     4.10.3          1.10.3          Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx     4.10.2          1.10.2          Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx     4.10.1          1.10.1          Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx     4.10.0          1.10.0          Ingress controller for Kubernetes using NGINX a...

# helm 下載指定版本：4.11.2 
helm fetch ingress-nginx/ingress-nginx --version 4.11.2 

# 解壓ingress-nginx4.11.2 版本安裝包
tar -xvf ingress-nginx-4.11.2.tgz

# 編輯 ingress-nginx 配置 values.yaml
vi ingress-nginx/values.yaml 

# 配置controller鏡像
controller:
  image:
    chroot: false
    registry: registry.k8s.io
    image: ingress-nginx/controller
    tag: "v1.11.2"
    #digest: sha256:d5f8217feeac4887cb1ed21f27c2674e58be06bd8f5184cacea2a69abaf78dce
    # digest 需配置為空才能拉取 registry.k8s.io/ingress-nginx/controller:v1.11.2 鏡像
    digest:

# 配置admissionWebhooks鏡像
controller:
  admissionWebhooks:
    patch:
      image:
        digest: sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3
        # digest 需配置為空才能拉取 registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3 鏡像
        digest:

# ingress 暴露為NodePort
controller:
  service:
    #type: LoadBalancer
    type: NodePort
# k8s 拉取鏡像倉庫驗(yàn)證secret
#imagePullSecrets: []
imagePullSecrets: 
- name: scr

# helm chart 安裝（存在就更新） ingress-nginx  
cd /root/ingress-nginx && helm upgrade --install ingress-nginx .  --namespace ingress-nginx --create-namespace 

# 卸載 ingress-nginx 
helm uninstall ingress-nginx -n ingress-nginx
# 查看安裝 ingress 
helm list -A|grep ingress
# 驗(yàn)證ingress 組件狀態(tài)，是否正常拉起
kubectl get svc -A |grep ingress
kubectl get pod -A |grep ingress
kubectl get deploy -n ingress-nginx               ingress-nginx-controller   -oyaml

# ingress-nginx 成功安裝效果
[root@bj-arm-master ingress-nginx]# helm upgrade --install ingress-nginx .  --namespace ingress-nginx --create-namespace 
Release "ingress-nginx" has been upgraded. Happy Helming!
NAME: ingress-nginx
LAST DEPLOYED: Fri Sep  6 11:20:42 2024
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
Get the application URL by running these commands:
  export HTTP_NODE_PORT=$(kubectl get service --namespace ingress-nginx ingress-nginx-controller --output jsonpath="{.spec.ports[0].nodePort}")
  export HTTPS_NODE_PORT=$(kubectl get service --namespace ingress-nginx ingress-nginx-controller --output jsonpath="{.spec.ports[1].nodePort}")
  export NODE_IP="$(kubectl get nodes --output jsonpath="{.items[0].status.addresses[1].address}")"

  echo "Visit http://${NODE_IP}:${HTTP_NODE_PORT} to access your application via HTTP."
  echo "Visit https://${NODE_IP}:${HTTPS_NODE_PORT} to access your application via HTTPS."

An example Ingress that makes use of the controller:
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: example
    namespace: foo
  spec:
    ingressClassName: nginx
    rules:
      - host: www.example.com
        http:
          paths:
            - pathType: Prefix
              backend:
                service:
                  name: exampleService
                  port:
                    number: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
      - hosts:
        - www.example.com
        secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

6.配置 ingress 轉(zhuǎn)發(fā)規(guī)則:

-- ingress配置demo1： 訪問 路徑/ 轉(zhuǎn)發(fā)到后端 namespace為default的 bte-service的8080端口
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: bte
 namespace: default
 #annotations:
   # 無論客戶端請求的是哪個(gè)路徑，Ingress 控制器都會將目標(biāo)請求路徑重寫為根路徑 /
   # 當(dāng)用戶訪問 http://example.com/foo 時(shí)，NGINX Ingress Controller 會將請求重寫為 http://my-service:80/。也就是說，任何通過 /foo 訪問的請求都會轉(zhuǎn)發(fā)到 my-service 服務(wù)，并且請求路徑會被重寫為根路徑 /
   #nginx.ingress.kubernetes.io/rewrite-target: /
spec:
 # 指定 Ingress Controller 的類型 為 nginx 類型：告訴 Kubernetes，這個(gè) Ingress 由 NGINX Ingress Controller 處理
 ingressClassName: nginx
 rules:
 #- host: "*"
 - http:
     paths:
     - path: /
       pathType: Prefix
       backend:
         service:
           name: bte-service
           port:
             number: 8080

-- ingress配置demo2:  訪問 路徑/layout 轉(zhuǎn)發(fā)到后端 namespace為default的 layout-service的8080端口
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: layout
 namespace: default
 annotations:
   # 無論客戶端請求的是哪個(gè)路徑，Ingress 控制器都會將目標(biāo)請求路徑重寫為根路徑 /
   # 當(dāng)用戶訪問 http://example.com/foo 時(shí)，NGINX Ingress Controller 會將請求重寫為 http://my-service:80/。也就是說，任何通過 /foo 訪問的請求都會轉(zhuǎn)發(fā)到 my-service 服務(wù)，并且請求路徑會被重寫為根路徑 /
   nginx.ingress.kubernetes.io/rewrite-target: /
spec:
 # k8s 1.18版本后 指定 Ingress Controller 的類型 為 nginx 類型配置：告訴 Kubernetes，這個(gè) Ingress 由 NGINX Ingress Controller 處理； 
 ingressClassName: nginx  
 rules:
 #- host: "*"
 - http:
     paths:
     - path: /layout
       pathType: Prefix
       backend:
         service:
           name: layout-service
           port:
             number: 8080

3.安裝過程遇到安裝失敗問題處理

安裝遇到問題: 無法正常拉取鏡像ImagePullBackOff

# kubectl get pod -A |grep ingress
ingress-nginx      ingress-nginx-admission-create-nz6hv       0/1     ImagePullBackOff   0          64s
# kubectl describe pod -n ingress-nginx      ingress-nginx-admission-create-nz6hv 

問題1: 報(bào)錯(cuò): 無法正常拉取鏡像 registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3@sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3

image.png

image.png

此鏡像版本比我們離線導(dǎo)入的image tag 多了 @sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3 信息

image.png

查看helm chart 源碼分析問題原因

無法正常拉取鏡像 registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.3@sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3
vi /root/ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml & job-patchWebhook.yaml
digest: sha256:a320a50cc91bd15fd2d6fa6de58bd98c1bd64b9a6f926ce23a600d87043455a3

image.png

image.png

** fix: digest 設(shè)置為空即可 **

image.png

重啟 ingress-nginx

helm upgrade --install ingress-nginx . --namespace ingress-nginx --create-namespace

問題2: 無法正常拉取鏡像 registry.k8s.io/ingress-nginx/controller:v1.11.2@sha256:d5f8217feeac4887cb1ed21f27c2674e58be06bd8f5184cacea2a69abaf78dce

# kubectl get pod -A |grep ingress
ingress-nginx      ingress-nginx-controller-5bddfb7dbf-gzjsx   0/1     ImagePullBackOff   0          49s     100.78.46.152    bj-arm-node1    <none>           <none>
# kubectl describe pod -n ingress-nginx      ingress-nginx-controller-5bddfb7dbf-gzjsx
  Failed to pull image "registry.k8s.io/ingress-nginx/controller:v1.11.2@sha256:d5f8217feeac4887cb1ed21f27c2674e58be06bd8f5184cacea2a69abaf78dce": Error response from daemon: Get "https://registry.k8s.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

此鏡像版本比我們離線導(dǎo)入的image tag 多了 @sha256:d5f8217feeac4887cb1ed21f27c2674e58be06bd8f5184cacea2a69abaf78dce 信息

查看helm chart 源碼分析問題原因

cat /root/ingress-nginx/templates/controller-deployment.yaml

image.png

###### 檢查當(dāng)前目錄下所有yaml文件中包含 字符串 ingress-nginx.imageDigest
grep -o "ingress-nginx.imageDigest"  ./*.*
# 輸出 
/root/ingress-nginx/templates/_helpers.tpl 

image.png

image.png

image.png

fix: digest 設(shè)置為空即可

image.png

重啟 ingress-nginx

helm upgrade --install ingress-nginx .  --namespace ingress-nginx --create-namespace 

ingress-nginx pod正常拉起，問題fix

# kubectl get pod -A |grep ingress
ingress-nginx      ingress-nginx-controller-785fcc99b-2zdhx   1/1     Running   0          22s

image.png

image.png

-- 問題fix!

參考文檔

https://kubernetes.io/docs/concepts/services-networking/ingress/ ingress文檔
https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/ Ingress Controllers 文檔
-- ingress-nginx相關(guān)文檔
https://github.com/kubernetes/ingress-nginx/blob/main/README.md#readme ingres nginx controller github文檔
https://github.com/kubernetes/ingress-nginx Ingress-nginx 文檔(支持 helm chart部署) use
https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx ingress-nginx helm chart文檔
https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx#values ingress-nginx helm chart values.yaml 配置說明
https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/index.md
https://kubernetes.github.io/ingress-nginx/user-guide/tls/ ngress-nginx 配置文檔