1 基于TLS的通信安全配置

etcd通過簡單的配置就能支持基于https的安全通信。https通信需要準備相應的證書文件。本文通過cfssl工具生成需要的證書文件

1.1 cfssl工具安裝

cfssl提供了一系列與證書管理相關的工具，根據需要下載相應的二進制文件即可。幾個常用的工具包括：
cfssl_1.6.1_linux_amd64、cfssljson_1.6.1_linux_amd64、cfssl-certinfo_1.6.1_linux_amd64

# cp cfssl_1.6.1_linux_amd64 /usr/bin/cfssl
# cp cfssljson_1.6.1_linux_amd64 /usr/bin/cfssljson
# cp cfssl-certinfo_1.6.1_linux_amd64 /usr/bin/cfssl-certinfo
# chmod a+x /usr/bin/cfssl && chmod a+x /usr/bin/cfssljson && chmod a+x /usr/bin/cfssl-certinfo

1.2 生成CA（根）證書

生成一個自簽名的CA證書，作為整個集群的根證書。其他的所有證書均由該根證書簽發

創建一個根證書的CSR（Certificate Signing Request）文件ca-csr.json，這樣需要生成CA證書時相應的配置可以重復使用。

$ cd ssl
$ mkdir ca && cd ca

$ vim ca-csr.json
{
    "CN": "Self-Signed-CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "China",
            "ST": "Guangdong",
            "L": "Shenzhen",
            "O": "sys",
            "OU": "sys"
        }
    ]
}

$ ls
ca-csr.json

CSR文件提供了需要認證的信息。CN（Common Name）是證書公共名字，一般是證書申請者的標識，如網站的話一般是網站域名。這里因為是自簽名證書，自己取一個標識名字就可以了。key字段是生成的密鑰的算法及相關配置。names字段指定了申請者的更詳細的信息。C（Country）：國家；ST（State）：省（州）；L（Location）：城市；O（Organization）：公司等組織；OU（Organization Unit）：公司等組織內的子單位。因為是根證書CSR，所有不需要提供hosts字段。

證書中的CN（Common Name）和O（Organization Name）是兩個需要特別關注的字段。CN字段對于 SSL 證書，一般為網站域名；而對于代碼簽名證書則為申請單位名稱；而對于客戶端證書則為證書申請者的姓名；在k8s系統中，kube-apiserver 從證書中提取CN字段作為請求的用戶名（User Name）；瀏覽器使用該字段驗證網站是否合法；O字段對于 SSL 證書，一般為網站域名；而對于代碼簽名證書則為申請單位名稱；而對于客戶端單位證書則為證書申請者所在單位名稱；在k8s系統中，kube-apiserver 從證書中提取該字段作為請求用戶所屬的組（Group）。

執行命令生成根證書：

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2022/08/17 21:19:59 [INFO] generating a new CA key and certificate from CSR
2022/08/17 21:19:59 [INFO] generate received request
2022/08/17 21:19:59 [INFO] received CSR
2022/08/17 21:19:59 [INFO] generating key: rsa-2048
2022/08/17 21:20:00 [INFO] encoded CSR
2022/08/17 21:20:00 [INFO] signed certificate with serial number 46204290406799778272816736364379807158759070127

$ ls
ca.csr  ca-csr.json  ca-key.pem  ca.pem

命令執行完后，生成了三個文件：ca-key.pem、ca.pem、ca.csr。ca-key.pem是CA證書私鑰，為其他申請者簽發證書時，CA使用這個私鑰加密申請的證書認證信息。ca.pem是CA證書，里面包含了簽名的CA公鑰，需要對證書信息進行驗證的使用者使用該證書中的公鑰解密認證信息，從而對證書進行認證、ca.csr是編碼后的CSR信息。

解析一下證書認證內容，可以發現就是上面CSR中填寫的信息

$ cfssl-certinfo -cert ca.pem
{
  "subject": {
    "common_name": "Self-Signed-CA",
    "country": "China",
    "organization": "sys",
    "organizational_unit": "sys",
    "locality": "Shenzhen",
    "province": "Guangdong",
    "names": [
      "China",
      "Guangdong",
      "Shenzhen",
      "sys",
      "sys",
      "Self-Signed-CA"
    ]
  },
  "issuer": {
    "common_name": "Self-Signed-CA",
    "country": "China",
    "organization": "sys",
    "organizational_unit": "sys",
    "locality": "Shenzhen",
    "province": "Guangdong",
    "names": [
      "China",
      "Guangdong",
      "Shenzhen",
      "sys",
      "sys",
      "Self-Signed-CA"
    ]
  },
  "serial_number": "46204290406799778272816736364379807158759070127",
  "not_before": "2022-08-17T13:15:00Z",
  "not_after": "2027-08-16T13:15:00Z",
  "sigalg": "SHA256WithRSA",
  "authority_key_id": "A:13:B:5B:AE:FB:D0:CA:1E:BF:49:6:D2:71:FD:29:9E:8E:3D:BE",
  "subject_key_id": "A:13:B:5B:AE:FB:D0:CA:1E:BF:49:6:D2:71:FD:29:9E:8E:3D:BE",
  "pem": "-----BEGIN CERTIFICATE-----\nMIIDzDCCArSgAwIBAgIUCBffPeRx9Y4faKqYFL0Nj3ssGa8wDQYJKoZIhvcNAQEL\nBQAwbDEOMAwGA1UEBhMFQ2hpbmExEjAQBgNVBAgTCUd1YW5nZG9uZzERMA8GA1UE\nBxMIU2hlbnpoZW4xDDAKBgNVBAoTA3N5czEMMAoGA1UECxMDc3lzMRcwFQYDVQQD\nEw5TZWxmLVNpZ25lZC1DQTAeFw0yMjA4MTcxMzE1MDBaFw0yNzA4MTYxMzE1MDBa\nMGwxDjAMBgNVBAYTBUNoaW5hMRIwEAYDVQQIEwlHdWFuZ2RvbmcxETAPBgNVBAcT\nCFNoZW56aGVuMQwwCgYDVQQKEwNzeXMxDDAKBgNVBAsTA3N5czEXMBUGA1UEAxMO\nU2VsZi1TaWduZWQtQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCs\nSBNyzo/mQcKoE2LJ9H/Q0Ynt5ZAmqFRfJs0Zvi0g/podzihgwH4BqW86pwhTbRTn\nnVKkPX5kFuf3/EBtWTZHYCIiA/54oKITYaCvEg6OmZmGWfxbFWCoNz+7zSSkr7K2\nqfbfR8wuRj0mbAgY+lrypypdSP1jzdu/cygLywtKzV7KEDC9X3+fUo9B5NYYZrDo\nysTPnSaoUAmRMSxvUCyL28dAmP1WaGFVcz1n/CPk//ENoHifOf/qRd/OeV+8WI35\nhtkArI3plCrnU4XfRqwu9/lBdzXgy4SAk7QkEEfnF6+GkOxUFawQV5PDfvBMCUAN\nG2eWKV77nE/UK7NJxi9zAgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMB\nAf8ECDAGAQH/AgECMB0GA1UdDgQWBBQKEwtbrvvQyh6/SQbScf0pno49vjAfBgNV\nHSMEGDAWgBQKEwtbrvvQyh6/SQbScf0pno49vjANBgkqhkiG9w0BAQsFAAOCAQEA\npaJsh/i68SDSsGU2oC73O7nxl9L/MawjU/ckDgrG01pCg6mRO3KWot+xcajGf6Ur\nrFwAba7o8Oh7CaRNovPkidOPSh24aK3b0C+gpmv88gXjOrum7FvWMRrImF6E1YiU\n+n+WvSJNA7Geok5KjKfpghghQREcc5Vipdtgtofr4NFIQGWw1dhF1h7MoktH3sII\n8Q6z8w1mjS/iZ8W38nfYNk0MK6Q2KosUcTAwYHaYfCrQTW7fg6rbDX1vs3GQpjlp\nIQtDBjR/6ndsxc4ZJoaKvPaGLBVBXNjtgH+EksKCWIYdeU9RO56watPbecLvoS8E\njVIGXeZBuza45mDLF030Aw==\n-----END CERTIFICATE-----\n"
}

1.3 etcd服務生成相關證書

1.3.1 創建證書配置

由于證書生成過程中一些公共配置會反復用到，所以可以配置到一個公共配置文件中，待后續使用

$ cd ssl
$ vim cert-config.json
{
    "signing": {
        "default": {
            "expiry": "8760h"
        },
        "profiles": {
            "server": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

配置文件中expiry字段規定了生成的證書的有效時間，上面默認是1年。配置文件中還定義了三個profile，使用時可以指定相應的profile，從而使用相應profile中的配置。usage字段配置了生成的證書可以用在哪些方面：比如簽名（signing）、加密（key encipherment）、客戶端對服務端進行認證（server auth）、服務端對客戶端進行認證（client auth）。

1.3.2 生成etcd集群間通信使用的證書

創建相應的CSR文件，hosts字段填寫集群節點相互用來通信的ip地址和dns名字等。建立tls連接時會檢查實際報文的源ip地址等是否在證書的hosts字段中

$ cd ssl
$ mkdir etcd-peer && cd etcd-peer
$ vim etcd-peer-csr.json

{
    "CN": "etcd",
    "hosts": [
      "127.0.0.1",
      "10.131.21.11",
      "10.131.21.12",
      "10.131.21.13",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Shenzhen",
            "L": "Shenzhen",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

$ ls
etcd-peer-csr.json

# ca.pem和ca.pem是上面生成的CA根證書和密鑰，配置信息使用cert-config.json中的peer profile。生成的文件名為etcd-peer
$ cfssl gencert -ca ../ca/ca.pem -ca-key ../ca/ca-key.pem -config ../cert-config.json -profile peer etcd-peer-csr.json | cfssljson -bare etcd-peer -
2022/08/17 21:26:54 [INFO] generate received request
2022/08/17 21:26:54 [INFO] received CSR
2022/08/17 21:26:54 [INFO] generating key: rsa-2048
2022/08/17 21:26:54 [INFO] encoded CSR
2022/08/17 21:26:54 [INFO] signed certificate with serial number 322003155427865617191473591973409438172195496664
2022/08/17 21:26:54 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

$ ls
etcd-peer.csr  etcd-peer-csr.json  etcd-peer-key.pem  etcd-peer.pem

1.3.3 生成etcd節點與client通信使用的證書

跟生成peer證書類似，只是CSR中的hosts只需要包括節點本身用于通信的ip地址即可。由于CN和O字段的特殊用途，根據需要設置為對應的值，我這里沒有改。profile使用server profile。

$ cd ssl
$ mkdir etcd-server && cd etcd-server
$ vim etcd-server-csr.json

{
    "CN": "etcd",
    "hosts": [
      "127.0.0.1",
      "10.131.21.11"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Shenzhen",
            "L": "Shenzhen",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

$ ls
etcd-server-csr.json

$ cfssl gencert -ca ../ca/ca.pem -ca-key ../ca/ca-key.pem -config ../cert-config.json -profile server etcd-server-csr.json | cfssljson -bare etcd-server -

$ ls
etcd-server.csr  etcd-server-csr.json  etcd-server-key.pem  etcd-server.pem

1.3.4 為客戶端生成證書

假設客戶端節點的ip地址為10.131.21.14，用與上面類似的方式生成證書。hosts字段包括自己用于通信的ip，由于CN和O字段的特殊用途，根據需要設置為對應的值，我這里沒有改。profile使用client profile。

$ cd ssl
$ mkdir etcd-client && cd etcd-client
$ vim etcd-client-csr.json

{
    "CN": "etcd",
    "hosts": [
      "127.0.0.1",
      "10.131.21.14"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Shenzhen",
            "L": "Shenzhen",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

$ ls
etcd-client-csr.json

$ cfssl gencert -ca ../ca/ca.pem -ca-key ../ca/ca-key.pem -config ../cert-config.json -profile client etcd-client-csr.json | cfssljson -bare etcd-client -

$ ls
etcd-client.csr  etcd-client-csr.json  etcd-client-key.pem  etcd-client.pem

1.3.5 生成的證書目錄結構

$ tree ssl
ssl
├── ca
│   ├── ca.csr
│   ├── ca-csr.json
│   ├── ca-key.pem
│   └── ca.pem
├── cert-config.json
├── etcd-client
│   ├── etcd-client.csr
│   ├── etcd-client-csr.json
│   ├── etcd-client-key.pem
│   └── etcd-client.pem
├── etcd-peer
│   ├── etcd-peer.csr
│   ├── etcd-peer-csr.json
│   ├── etcd-peer-key.pem
│   └── etcd-peer.pem
└── etcd-server
    ├── etcd-server.csr
    ├── etcd-server-csr.json
    ├── etcd-server-key.pem
    └── etcd-server.pem

在etcd節點的部署工作目錄下創建一個ssl目錄，并將上面ca、etcd-peer、etcd-server目錄下的所有證書相關文件拷貝到相應節點的ssl目錄下。
例如，節點10.131.21.11的ssl目錄下內容為：

$ cd etcd
$ ls ssl
ca.pem etcd-peer-key.pem etcd-peer.pem etcd-server-key.pem etcd-server.pem

1.4 啟動集群

假設集群有三個節點，服務ip分別為10.131.21.11、10.131.21.12、10.131.21.13，為這三個節點分別創建相應的配置文件

1.4.1 節點配置

節點1（10.131.21.11）配置文件內容：

$ cat conf/etcd.yaml

# 節點名字，集群內唯一
name: etcd-cnlab0

# 數據存儲目錄
data-dir: ./data

# 日志文件配置
# logger: zap
log-outputs: ['./log/etcd.log']
# log-level: info

# 客戶端連接相關配置
# 服務監聽端點
listen-client-urls: https://0.0.0.0:2379
# 向客戶端發布的服務端點
advertise-client-urls: https://10.131.21.11:2379

# 集群相關配置
# 監聽集群其他節點連接的端點
listen-peer-urls: https://0.0.0.0:2380
# 向集群其他節點發布的服務端點
initial-advertise-peer-urls: https://10.131.21.11:2380
# 集群成員的名字以及服務端點列表，名字與每個節點配置的name字段值對應
initial-cluster: etcd-cnlab0=https://10.131.21.11:2380,etcd-cnlab1=https://10.131.21.12:2380,etcd-cnlab2=https://10.131.21.13:2380
# 集群標識token，可以認為是集群名字
initial-cluster-token: etcd-cnlab
# 創建一個新的集群。如果data-dir目錄下的數據屬于另外一個集群，則無法啟動
inital-cluster-state: new

# 配置證書
# 用于客戶端認證etcd的證書
client-transport-security:
  cert-file: ./ssl/etcd-server.pem
  key-file: ./ssl/etcd-server-key.pem
  trusted-ca-file: ./ssl/ca.pem
  client-cert-auth: true
# 用于集群節點間相互認證的證書
peer-transport-security:
  cert-file: ./ssl/etcd-peer.pem
  key-file: ./ssl/etcd-peer-key.pem
  trusted-ca-file: ./ssl/ca.pem
  client-cert-auth: true

節點2（10.131.21.12）配置文件內容：

$ cat conf/etcd.yaml

name: etcd-cnlab1
data-dir: ./data
listen-client-urls: https://0.0.0.0:2379
advertise-client-urls: https://10.131.21.12:2379
listen-peer-urls: https://0.0.0.0:2380

initial-advertise-peer-urls: https://10.131.21.12:2380
initial-cluster: etcd-cnlab0=https://10.131.21.11:2380,etcd-cnlab1=https://10.131.21.12:2380,etcd-cnlab2=https://10.131.21.13:2380
initial-cluster-token: etcd-cnlab
inital-cluster-state: new

client-transport-security:
  cert-file: ./ssl/etcd-server.pem
  key-file: ./ssl/etcd-server-key.pem
  trusted-ca-file: ./ssl/ca.pem
  client-cert-auth: true
peer-transport-security:
  cert-file: ./ssl/etcd-peer.pem
  key-file: ./ssl/etcd-peer-key.pem
  trusted-ca-file: ./ssl/ca.pem
  client-cert-auth: true

節點3（10.131.21.13）配置文件內容：

$ cat conf/etcd.yaml

name: etcd-cnlab2
data-dir: ./data
listen-client-urls: https://0.0.0.0:2379
advertise-client-urls: https://10.131.21.13:2379
listen-peer-urls: https://0.0.0.0:2380

initial-advertise-peer-urls: https://10.131.21.13:2380
initial-cluster: etcd-cnlab0=https://10.131.21.11:2380,etcd-cnlab1=https://10.131.21.12:2380,etcd-cnlab2=https://10.131.21.13:2380
initial-cluster-token: etcd-cnlab
inital-cluster-state: new

client-transport-security:
  cert-file: ./ssl/etcd-server.pem
  key-file: ./ssl/etcd-server-key.pem
  trusted-ca-file: ./ssl/ca.pem
  client-cert-auth: true
peer-transport-security:
  cert-file: ./ssl/etcd-peer.pem
  key-file: ./ssl/etcd-peer-key.pem
  trusted-ca-file: ./ssl/ca.pem
  client-cert-auth: true

1.4.2 啟動三個節點

$ ./bin/etcd --config-file=conf/etcd.yaml

1.5 客戶端連接etcd

在客戶端節點10.131.20.14上連接集群節點10.131.20.11，并執行命令

$ ./bin/etcdctl --endpoints=https://10.131.20.11:2379 --cert=./ssl/etcd.pem --key=./ssl/etcd-key.pem --cacert=./ssl/ca.pem member list
5adcdda313f6e536, started, etcd-cnlab1, https://10.131.20.12:2380, https://10.131.20.12:2379, false
886dc7c192c66fe4, started, etcd-cnlab2, https://10.131.20.13:2380, https://10.131.20.13:2379, false
fa92ba5824e127ae, started, etcd-cnlab0, https://10.131.20.11:2380, https://10.131.20.11:2379, false

2 基于角色的權限認證

不管是否開啟TLS，etcd都可以通過用戶名（user）和角色（role）對用戶的訪問權限進行控制。

默認情況下，etcd集群沒有開啟基于角色的權限認證。開啟權限認證，一定存在一個root用戶和root角色，root角色自然擁有所有權限。所以開啟權限認證，需要先創建root用戶和root角色，并將root用戶綁定到root角色。

在任意一個集群節點上執行下列命令，創建root用戶和root角色，并綁定。

# 查詢已配置的用戶和角色，默認沒有
$ ./bin/etcdctl user list
$ ./bin/etcdctl role list

# 創建root用戶和root角色
$ ./bin/etcdctl user add root
$ ./bin/etcdctl role add root

$ ./bin/etcdctl user list
root
$ ./bin/etcdctl role list
root

# 綁定角色
$ ./bin/etcdctl user grant-role root root

在任意一個集群節點上執行下列命令，開始權限認證：

$ ./bin/etcdctl auth enable

現在開始，需要使用用戶名和密碼才能訪問etcd集群了

$ ./bin/etcdctl --user=root:etcd123 role list
root

創建非root用戶和角色

$ ./bin/etcdctl --user=root:etcd123 user add u_sys

$ ./bin/etcdctl --user=root:etcd123 role add r_sys

# key的讀權限
$ ./bin/etcdctl role grant-permission r_sys read /foo

# 整個目錄的寫權限
$ ./bin/etcdctl role grant-permission r_sys write /sys/*

# 讀寫權限
$ ./bin/etcdctl role grant-permission r_sys readwrite /abc

# 查看角色的權限
$ ./bin/etcdctl role get r_sys
Role r_sys
KV Read:
    /abc
    /foo
KV Write:
    /abc
    [/sys/block, /sys/bus)

# 綁定用戶和角色，使用戶獲得相應的權限
$ ./bin/etcdctl user grant-role u_sys r_sys

# 撤銷角色的權限
$ ./bin/etcdctl role revoke-permission r_sys /foo

刪除命令

$ ./bin/etcdctl role delete r_sys
$ ./bin/etcdctl user delete u_sys