springcloud + apollo的微服務化部署

創建namespace

建立屬于微服務體系下的namespace,cloud.yml

apiVersion: v1
kind: Namespace
metadata:
  name: springcloud

執行 kubectl apply -f cloud.yml

eureka

由于eureka是有狀態的服務，因此采用statefulset部署

1. 首先創建鏡像,編寫DOCKFILE

FROM java:8
copy ./www /data/www
entrypoint java ${JVM:=-Xms1024m -Xmx1024m} -Djava.security.egd=file:/dev/./urandom -Dfile.encoding=UTF8 -jar /data/www/iggreport-registry-2.1.5.RELEASE.jar ${APP_OPT}

執行docker build -t eureka:v1 .

2. 創建eureka.yml

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  namespace: springcloud
type: Opaque
data:
  password: WDhHbkdYcVdGUlg5V3Z3ZQ== 

---
apiVersion: v1
kind: Service
metadata:
  name: eureka
  namespace: springcloud
  labels:
    app: eureka
spec:
  clusterIP: None
  ports:
    - port: 8761
      targetPort: 8761
  selector:
    app: eureka

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: eureka-set
  namespace: springcloud
spec:
  serviceName: eureka
  replicas: 2
  selector:
    matchLabels:
      app: eureka
  template:
    metadata:
      labels:
        app: eureka
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: eureka
          image: eureka:v4
          ports:
          - containerPort: 8761
          env:
            - name: APP_NAME
              value: "eureka"
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: SPRING_APPLICATION_NAME
              value: "igg-report-registry" 
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysecret
                  key: password
            - name: APP_OPT
              value: "
                     --eureka.instance.hostname=${POD_NAME}.${APP_NAME}
                     --registerWithEureka=true
                     --fetchRegistry=true
                     --eureka.instance.preferIpAddress=false
                     --eureka.client.serviceUrl.defaultZone=http://${SPRING_APPLICATION_NAME}:${PASSWORD}@eureka-set-0.${APP_NAME}:8761/eureka/,http://${SPRING_APPLICATION_NAME}:${PASSWORD}@eureka-set-1.${APP_NAME}:8761/eureka/
                     "

由于我的注冊中心有做安全校驗，因此配置注冊中心的時候需要填寫密碼，而密碼不想明文寫入到配置的話需要自己加密下,加密方法如下:

[root@master eureka]# echo -n 'X8GnGXqWFRX9Wvwe' | base64
WDhHbkdYcVdGUlg5V3Z3ZQ==

"WDhHbkdYcVdGUlg5V3Z3ZQ=="這個就是經過base64加密后的密碼了。
StatefulSet 作為 Controller 為 Pod 提供唯一的標識。它可以保證部署和 scale 的順序。
StatefulSet是為了解決有狀態服務的問題（對應Deployments和ReplicaSets是為無狀態服務而設計），其應用場景包括：

• 穩定的持久化存儲，即Pod重新調度后還是能訪問到相同的持久化數據，基于PVC來實現
• 穩定的網絡標志，即Pod重新調度后其PodName和HostName不變，基于Headless Service（即沒有Cluster IP的Service）來實現
• 有序部署，有序擴展，即Pod是有順序的，在部署或者擴展的時候要依據定義的順序依次依次進行（即從0到N-1，在下一個Pod運行之前所有之前的Pod必須都是Running和Ready狀態），基于init containers來實現
• 有序收縮，有序刪除（即從N-1到0）
  從上面的應用場景可以發現，StatefulSet由以下幾個部分組成：
• 用于定義網絡標志（DNS domain）的Headless Service
• 用于創建PersistentVolumes的volumeClaimTemplates
• 定義具體應用的StatefulSet
  StatefulSet中每個Pod的DNS格式為statefulSetName-{0..N-1}.serviceName.namespace.svc.cluster.local，其中
• serviceName為Headless Service的名字
• 0..N-1為Pod所在的序號，從0開始到N-1
• statefulSetName為StatefulSet的名字
• namespace為服務所在的namespace，Headless Servic和StatefulSet必須在相同的namespace
• .cluster.local為Cluster Domain

apollo配置中心

由于apollo配置中心需要mysql數據庫的支持，而mysql不是部署在k8s集群內，因此需要建立一個service可以代理到外部服務去
endpoints.yml

kind: Service
apiVersion: v1
metadata:
  name: my-service
  namespace: springcloud
spec:
  clusterIP: 192.168.186.163
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306

---
kind: Endpoints
apiVersion: v1
metadata:
  name: my-service
  namespace: springcloud
subsets:
  - addresses:
      - ip: 10.0.3.166
    ports:
      - port: 3306

這樣通過訪問這個serviceIp的3306就能訪問到外部節點的mysql

apollo-config部署

先自建apollo-config的鏡像

FROM java:8
COPY www /data/www
entrypoint java ${JVM:=-Xms1024m -Xmx1024m} -Djava.security.egd=file:/dev/./urandom -Dfile.encoding=UTF8 -jar /data/www/apollo-configservice-1.5.1.jar ${APP_OPT}

apollo-config.yml

apiVersion: v1
kind: Secret
metadata:
  name: apollo-config
  namespace: springcloud
type: Opaque
data:
  password: QXF1YXl3RGJDYVNoeEczbA== 

---
apiVersion: v1
kind: Service
metadata:
  name: apollo-config
  namespace: springcloud
  labels:
    app: apollo-config
spec:
  ports:
    - port: 8888
      targetPort: 8888
  selector:
    app: apollo-config

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apollo-config
  namespace: springcloud
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apollo-config
  template:
    metadata:
      labels:
        app: apollo-config
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: apollo-config
          image: apollo-config:v4
          ports:
          - containerPort: 8888
          env:
            - name: SPRING_APPLICATION_NAME
              value: "apollo-configservice" 
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  name: apollo-config
                  key: password
            - name: APP_OPT
              value: "
                     --eureka.instance.hostname=apollo-config
                     --registerWithEureka=true
                     --fetchRegistry=true
                     --eureka.instance.preferIpAddress=false
                     --spring.datasource.url=jdbc:mysql://my-service:3306/apollo_config?useSSL=false&characterEncoding=utf-8
                     --eureka.client.serviceUrl.defaultZone=http://${SPRING_APPLICATION_NAME}:${PASSWORD}@eureka-set-0:8761/eureka/,http://${SPRING_APPLICATION_NAME}:${PASSWORD}@eureka-set-1:8761/eureka/
                     "

apollo-admin部署

自建apollo-admin鏡像

FROM java:8
copy ./www /data/www
entrypoint java ${JVM:=-Xms1024m -Xmx1024m} -Djava.security.egd=file:/dev/./urandom -Dfile.encoding=UTF8 -jar /data/www/apollo-adminservice-1.5.1.jar ${APP_OPT}

apollo-admin.yml

apiVersion: v1
kind: Secret
metadata:
  name: apollo-admin
  namespace: springcloud
type: Opaque
data:
  password: QXF1YXl3RGJDYVNoeEczbA== 

---
apiVersion: v1
kind: Service
metadata:
  name: apollo-admin
  namespace: springcloud
  labels:
    app: apollo-admin
spec:
  ports:
    - port: 8070
      targetPort: 8070
  selector:
    app: apollo-admin

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apollo-admin
  namespace: springcloud
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apollo-admin
  template:
    metadata:
      labels:
        app: apollo-admin
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: apollo-admin
          image: apollo-admin:v1
          ports:
          - containerPort: 8070
          env:
            - name: APP_NAME
              value: apollo-admin
            - name: SPRING_APPLICATION_NAME
              value: "apollo-adminservice" 
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  name: apollo-admin
                  key: password
            - name: APP_OPT
              value: "
                     --eureka.instance.hostname=${APP_NAME}
                     --registerWithEureka=true
                     --fetchRegistry=true
                     --spring.datasource.url=jdbc:mysql://my-service:3306/apollo_config?useSSL=false&characterEncoding=utf-8
                     --eureka.instance.preferIpAddress=false
                     --eureka.client.serviceUrl.defaultZone=http://${SPRING_APPLICATION_NAME}:${PASSWORD}@eureka-set-0.${APP_NAME}:8761/eureka/,http://${SPRING_APPLICATION_NAME}:${PASSWORD}@eureka-set-1.${APP_NAME}:8761/eureka/
                     "

apollo-portal部署

自建apollo-portal鏡像

FROM java:8
copy ./www /data/www
entrypoint java ${JVM:=-Xms1024m -Xmx1024m} -Djava.security.egd=file:/dev/./urandom -Dfile.encoding=UTF8 -jar /data/www/apollo-portal-1.5.1.jar

apollo-portal.yml

apiVersion: v1
kind: Service
metadata:
  name: apollo-portal
  namespace: springcloud
  labels:
    app: apollo-portal
spec:
  ports:
    - port: 8060
      targetPort: 8060
  selector:
    app: apollo-portal

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apollo-portal
  namespace: springcloud
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apollo-portal
  template:
    metadata:
      labels:
        app: apollo-portal
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: apollo-portal
          image: apollo-portal:v1
          ports:
          - containerPort: 8060

由于apollo-portal需要web頁面對外訪問，因此需要使用ingress配合service，將其對外訪問

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-test
  namespace: springcloud
spec:
  rules:
    #定義域名
    - host: apollo.ingress.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: apollo-portal
              port:
                number: 8060

api-common部署

api-common有使用到apollo的配置中心
所以配置文件如下:

app:
  id: iggreport-api-common
apollo:
  meta: http://apollo-config:8888
  bootstrap:
    enabled: true
    namespaces: application,application.yml

構建鏡像Dockerfile

FROM java:8
copy ./www /data/www
entrypoint java ${JVM:=-Xms1024m -Xmx1024m} -Djava.security.egd=file:/dev/./urandom -Dfile.encoding=UTF8 -jar /data/www/iggreport-api-common-2.1.4.RELEASE.jar ${APP_OPT}

api.yml

apiVersion: v1
kind: Service
metadata:
  name: api-common-service
  namespace: springcloud
  labels:
    app: api-common-service
spec:
  ports:
    - port: 8090
      targetPort: 8090
  selector:
    app: api-common
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-common
  namespace: springcloud
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api-common
  template:
    metadata:
      labels:
        app: api-common
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: api-common
          image: api:v1
          ports:
          - containerPort: 8090
          env:
            - name: APP_NAME
              value: api-common-service
            - name: APP_OPT
              value: "
                     --eureka.instance.hostname=${APP_NAME}  #這里記得要用service的名字，否則網關使用此hostname無法訪問
                     "

gateway部署

由于網關需要redis,因此也需要建個service代理到外部服務的redis集群
endpoint.yml

kind: Service
apiVersion: v1
metadata:
  name: redis-service
  namespace: springcloud
spec:
  ports:
    - protocol: TCP
      port: 7000
      targetPort: 7000

---
kind: Endpoints
apiVersion: v1
metadata:
  name: redis-service
  namespace: springcloud
subsets:
  - addresses:
      - ip: 10.0.3.163
      - ip: 10.0.3.164
      - ip: 10.0.3.165
    ports:
      - port: 7000

自建鏡像

FROM java:8
copy ./www /data/www
entrypoint java ${JVM:=-Xms1024m -Xmx1024m} -Djava.security.egd=file:/dev/./urandom -Dfile.encoding=UTF8 -jar /data/www/iggreport-gateway-2.2.0.RELEASE.jar ${APP_OPT}

gateway.yml

apiVersion: v1
kind: Service
metadata:
  name: gateway-service
  namespace: springcloud
  labels:
    app: gateway-service
spec:
  ports:
    - port: 8080
      targetPort: 8080
  selector:
    app: gateway
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gateway
  namespace: springcloud
spec:
  replicas: 2
  selector:
    matchLabels:
      app: gateway
  template:
    metadata:
      labels:
        app: gateway
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: gateway
          image: gateway:v1
          ports:
          - containerPort: 8080

然后由于gateway需要對外，因此也需要加入到ingress管理

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-test
  namespace: springcloud
spec:
  rules:
    #定義域名
    - host: gateway.ingress.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: gateway-service
              port:
                number: 8080

這樣一個完整的springcloud + apollo的統一配置中心就基于k8s搭建完畢

[root@master k8s]# kubectl get po -o wide -n springcloud
NAME                             READY   STATUS    RESTARTS   AGE    IP           NODE   NOMINATED NODE   READINESS GATES
api-common-5c59b9b649-4rkrm      1/1     Running   0          116m   10.10.2.93   cdh2   <none>           <none>
api-common-5c59b9b649-swpq8      1/1     Running   0          116m   10.10.1.88   cdh3   <none>           <none>
apollo-admin-6dfcf44f8b-flwtc    1/1     Running   0          25h    10.10.2.83   cdh2   <none>           <none>
apollo-config-5d7bddf55c-cn4ph   1/1     Running   0          24h    10.10.2.90   cdh2   <none>           <none>
apollo-portal-bb7d4685d-srw2r    1/1     Running   0          24h    10.10.2.92   cdh2   <none>           <none>
eureka-set-0                     1/1     Running   0          24h    10.10.2.91   cdh2   <none>           <none>
eureka-set-1                     1/1     Running   0          24h    10.10.1.87   cdh3   <none>           <none>
gateway-949d4f958-4ft8x          1/1     Running   0          9m7s   10.10.1.89   cdh3   <none>           <none>
gateway-949d4f958-b4wc7          1/1     Running   0          9m7s   10.10.2.94   cdh2   <none>           <none>

[root@master k8s]# kubectl get svc -o wide -n springcloud
NAME                 TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)    AGE     SELECTOR
api-common-service   ClusterIP   192.168.144.76    <none>        8090/TCP   116m    app=api-common
apollo-admin         ClusterIP   192.168.118.145   <none>        8070/TCP   25h     app=apollo-admin
apollo-config        ClusterIP   192.168.134.51    <none>        8888/TCP   5d23h   app=apollo-config
apollo-portal        ClusterIP   192.168.31.38     <none>        8060/TCP   25h     app=apollo-portal
eureka               ClusterIP   None              <none>        8761/TCP   3d23h   app=eureka
gateway-service      ClusterIP   192.168.230.235   <none>        8080/TCP   9m27s   app=gateway
my-service           ClusterIP   192.168.186.163   <none>        3306/TCP   5h10m   <none>
redis-service        ClusterIP   192.168.143.180   <none>        7000/TCP   75m     <none>