tcp狀態(tài)機(jī)
上圖是 tcp 狀態(tài)流轉(zhuǎn)，建立連接時(shí)三次握手，斷開(kāi)時(shí)四次。搭建雙臂 full-nat 測(cè)試時(shí)發(fā)現(xiàn)，斷開(kāi)也通常是三次握手。并且 dpvs 流表的 tcp 狀態(tài)機(jī)和教科書(shū)的差別很大，具體表現(xiàn)為無(wú)論主動(dòng)還是被動(dòng)關(guān)閉，tcp 狀態(tài)都只會(huì)走到 TIME_WAIT, 而不是 LAST_ACK, 為什么呢？
測(cè)試環(huán)境
VIP: 202.108.10.1:6379
LIP: 192.168.168.7
RSIP: 192.168.168.2
CLIENT: 202.108.10.2
root@dpvs:~/dpvs# dpip addr show
inet 202.108.10.1/32 scope global dpdk0
valid_lft forever preferred_lft forever
inet 192.168.168.3/32 scope global dpdk1
valid_lft forever preferred_lft forever
inet 192.168.168.7/32 scope global dpdk1
valid_lft forever preferred_lft forever sa_used 1 sa_free 903151 sa_miss 0
root@dpvs:~/dpvs# ipvsadm -ln
IP Virtual Server version 0.0.0 (size=0)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 202.108.10.1:6379 wrr
-> 192.168.168.2:6379 FullNat 100 0 1
抓包分析
dpvs 編繹時(shí)一定打開(kāi) DEBUG 模式，日志也要調(diào)成 DEBUG 級(jí)別。客戶端發(fā)送 redis 命令并主動(dòng)斷開(kāi)連接
redis-cli -h 202.108.10.1 -p 6379 get set testclienthhhhhhhhhh testclienthhhhhhhhhh
客戶端抓包結(jié)果
22:11:13.362012 IP 202.108.10.2.33896 > 202.108.10.1.6379: Flags [S], seq 4164191644, win 29200, options [mss 1460,sackOK,TS val 125357496 ecr 0,nop,wscale 7], length 0
22:11:13.362834 IP 202.108.10.1.6379 > 202.108.10.2.33896: Flags [S.], seq 4072520983, ack 4164191645, win 29200, options [mss 1452,nop,nop,sackOK,nop,wscale 7], length 0
22:11:13.362877 IP 202.108.10.2.33896 > 202.108.10.1.6379: Flags [.], ack 1, win 229, length 0
22:11:13.362926 IP 202.108.10.2.33896 > 202.108.10.1.6379: Flags [P.], seq 1:77, ack 1, win 229, length 76
22:11:13.363255 IP 202.108.10.1.6379 > 202.108.10.2.33896: Flags [.], ack 77, win 229, length 0
22:11:13.363406 IP 202.108.10.1.6379 > 202.108.10.2.33896: Flags [P.], seq 1:51, ack 77, win 229, length 50
22:11:13.363437 IP 202.108.10.2.33896 > 202.108.10.1.6379: Flags [.], ack 51, win 229, length 0
22:11:13.363598 IP 202.108.10.2.33896 > 202.108.10.1.6379: Flags [F.], seq 77, ack 51, win 229, length 0
22:11:13.364140 IP 202.108.10.1.6379 > 202.108.10.2.33896: Flags [F.], seq 51, ack 78, win 229, length 0
22:11:13.364153 IP 202.108.10.2.33896 > 202.108.10.1.6379: Flags [.], ack 52, win 229, length 0
首先，202.108.10.2:33896 與 202.108.10.1:6379 進(jìn)行三次握手，然后發(fā)送數(shù)據(jù)。最后斷開(kāi)時(shí)是重點(diǎn)，只有三次揮手。后端 RS 202.108.10.1:6379 返回一個(gè) [F.] 包，也就是 fin + ack 包，相當(dāng)于原來(lái)四次揮手的二三步合并成一個(gè)包了。
服務(wù)端抓包結(jié)果
22:11:26.081295 IP 192.168.168.7.1028 > 192.168.168.2.6379: Flags [S], seq 2976553321, win 29200, options [exp-8468,mss 1460,sackOK,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,wscale 7], length 0
22:11:26.081325 IP 192.168.168.2.6379 > 192.168.168.7.1028: Flags [S.], seq 4072520983, ack 2976553322, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
22:11:26.081825 IP 192.168.168.7.1028 > 192.168.168.2.6379: Flags [.], ack 1, win 229, options [exp-8468], length 0
22:11:26.081858 IP 192.168.168.7.1028 > 192.168.168.2.6379: Flags [P.], seq 1:77, ack 1, win 229, options [exp-8468], length 76: RESP "get" "set" "testclienthhhhhhhhhh" "testclienthhhhhhhhhh"
22:11:26.081870 IP 192.168.168.2.6379 > 192.168.168.7.1028: Flags [.], ack 77, win 229, length 0
22:11:26.081990 IP 192.168.168.2.6379 > 192.168.168.7.1028: Flags [P.], seq 1:51, ack 77, win 229, length 50: RESP "ERR wrong number of arguments for 'get' command"
22:11:26.082308 IP 192.168.168.7.1028 > 192.168.168.2.6379: Flags [.], ack 51, win 229, length 0
22:11:26.082524 IP 192.168.168.7.1028 > 192.168.168.2.6379: Flags [F.], seq 77, ack 51, win 229, length 0
22:11:26.082624 IP 192.168.168.2.6379 > 192.168.168.7.1028: Flags [F.], seq 51, ack 78, win 229, length 0
22:11:26.083037 IP 192.168.168.7.1028 > 192.168.168.2.6379: Flags [.], ack 52, win 229, length 0
服務(wù)端也正常，三次握手，傳輸數(shù)據(jù)，最后同樣三次揮手斷開(kāi)連接。
dpvs 日志輸出
IPVS: conn lookup: [5] TCP 202.108.10.2:33896 -> 202.108.10.1:6379 miss
SAPOOL: sa_pool_fetch: 192.168.168.7:1028 fetched!
IPVS: new conn: [5] TCP 202.108.10.2:33896 202.108.10.1:6379 192.168.168.7:1028 192.168.168.2:6379 refs 2
IPVS: state trans: TCP in [S...] 202.108.10.2:33896->192.168.168.2:6379 state NONE->SYN_RECV conn.refcnt 2
IPVS: conn lookup: [5] TCP 192.168.168.2:6379 -> 192.168.168.7:1028 hit NEIGHBOUR: REACHABLE trans state to REACHABLE.
IPVS: conn lookup: [5] TCP 202.108.10.2:33896 -> 202.108.10.1:6379 hit
IPVS: state trans: TCP in [..A.] 202.108.10.2:33896->192.168.168.2:6379 state SYN_RECV->ESTABLISHED conn.refcnt 2
IPVS: conn lookup: [5] TCP 202.108.10.2:33896 -> 202.108.10.1:6379 hit
IPVS: conn lookup: [5] TCP 192.168.168.2:6379 -> 192.168.168.7:1028 hit
IPVS: conn lookup: [5] TCP 192.168.168.2:6379 -> 192.168.168.7:1028 hit
IPVS: conn lookup: [5] TCP 202.108.10.2:33896 -> 202.108.10.1:6379 hit
IPVS: conn lookup: [5] TCP 202.108.10.2:33896 -> 202.108.10.1:6379 hit
IPVS: state trans: TCP in [.FA.] 202.108.10.2:33896->192.168.168.2:6379 state ESTABLISHED->CLOSE_WAIT conn.refcnt 2
IPVS: conn lookup: [5] TCP 192.168.168.2:6379 -> 192.168.168.7:1028 hit
IPVS: state trans: TCP out [.FA.] 202.108.10.2:33896->192.168.168.2:6379 state CLOSE_WAIT->TIME_WAIT conn.refcnt 2
IPVS: conn lookup: [5] TCP 202.108.10.2:33896 -> 202.108.10.1:6379 hit
重點(diǎn)在 dpvs 日志，我們逐條分析。
- 客戶端 202.108.10.2:33896 發(fā)送 syn 請(qǐng)求建連，但是此時(shí)
conn lookup查找流表肯定不存在的，也就是第一條的 miss - 由于是雙臂模式，并且 fdir 的存在，必須在 LIP 上分配一個(gè)端口，那就是從 sa_pool 中分配 1028 lport
- new conn 真正的建立了連接五元組: proto, client, vip, lip, rs
- state trans 將連接由 state NONE 變成 SYN_RECV 狀態(tài)
- conn lookup hit 由后端 6379 返回的包命中了流表，然后 NEIGHBOUR 更新鄰居表。其實(shí)這次的包就是三次握手的第二個(gè) sync + ack
6-7. 完成 tcp 三次握手，狀態(tài)由 state SYN_RECV 變更成 ESTABLISHED
8-11. 這幾行都在雙向傳輸數(shù)據(jù)，命中了流表
12-13. 這兩行由 client 202.108.10.2:33896 發(fā)送 [.FA.] 主動(dòng)關(guān)閉連接。 流表狀態(tài)由 ESTABLISHED 變成了 CLOSE_WAIT
14-15. 這兩行由后端 rs 192.168.168.2:6379 發(fā)送了 [.FA.] 包，也就是四次揮手中的二三步合一起了。狀態(tài)由 CLOSE_WAIT 變成 TIME_WAIT
- 最后客戶端發(fā)送的 ack 包命中了流表
查看 ipvsadm
root@dpvs:~/dpvs# ipvsadm -ln -c
[5]tcp 7s TIME_WAIT 202.108.10.2:33896 202.108.10.1:6379 192.168.168.7:1028 192.168.168.2:6379
最后可以看到流表中連接一直處于 TIME_WAIT 狀態(tài)，直到超時(shí)被 conn_expire 刪除。上面的步聚是 client 主動(dòng)關(guān)閉，其實(shí)測(cè)試 rs 主動(dòng)關(guān)閉也是一樣的。
查看源碼
static struct tcp_state tcp_states[] = {
/* INPUT */
/* sNO, sES, sSS, sSR, sFW, sTW, sCL, sCW, sLA, sLI, sSA */
/*syn*/ {{sSR, sES, sES, sSR, sSR, sSR, sSR, sSR, sSR, sSR, sSR}},
/*fin*/ {{sCL, sCW, sSS, sTW, sTW, sTW, sCL, sCW, sLA, sLI, sTW}},
/*ack*/ {{sCL, sES, sSS, sES, sFW, sTW, sCL, sCW, sCL, sLI, sES}},
/*rst*/ {{sCL, sCL, sCL, sSR, sCL, sCL, sCL, sCL, sLA, sLI, sSR}},
/* OUTPUT */
/* sNO, sES, sSS, sSR, sFW, sTW, sCL, sCW, sLA, sLI, sSA */
/*syn*/ {{sSS, sES, sSS, sSR, sSS, sSS, sSS, sSS, sSS, sLI, sSR}},
/*fin*/ {{sTW, sFW, sSS, sTW, sFW, sTW, sCL, sTW, sLA, sLI, sTW}},
/*ack*/ {{sES, sES, sSS, sES, sFW, sTW, sCL, sCW, sLA, sES, sES}},
/*rst*/ {{sCL, sCL, sSS, sCL, sCL, sTW, sCL, sCL, sCL, sCL, sCL}},
/* INPUT-ONLY */
/* sNO, sES, sSS, sSR, sFW, sTW, sCL, sCW, sLA, sLI, sSA */
/*syn*/ {{sSR, sES, sES, sSR, sSR, sSR, sSR, sSR, sSR, sSR, sSR}},
/*fin*/ {{sCL, sFW, sSS, sTW, sFW, sTW, sCL, sCW, sLA, sLI, sTW}},
/*ack*/ {{sCL, sES, sSS, sES, sFW, sTW, sCL, sCW, sCL, sLI, sES}},
/*rst*/ {{sCL, sCL, sCL, sSR, sCL, sCL, sCL, sCL, sLA, sLI, sCL}},
};
ip_vs_proto_tcp.c 源碼狀態(tài)機(jī)的流轉(zhuǎn)圖，翻了下 lvs 代碼，其實(shí)是一樣的。INPUT 流，OUTPUT 流用于 nat fnat 相關(guān)的，INPUT-ONLY 服務(wù)于 dr tunnel 等等
小結(jié)
這么做其實(shí)是對(duì)的，無(wú)論 client 還是 rs 主動(dòng)關(guān)閉，總有一方會(huì)處于 time_wait 狀態(tài)，等待 2MSL 時(shí)間后再清除。做為 LB 只能等待時(shí)間更長(zhǎng)，才能兼容原有邏輯，做到包重傳。默認(rèn) dpvs time_wait 超時(shí)時(shí)間 7s，夠用了。
