問題描述
其他業務線的同學在測試環境發現應用程序一直不能獲取redis連接，我幫忙看了下。
首先看應用錯誤日志
Caused by: org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:97)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:143)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:41)
at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:85)
at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:55)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:169)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:149)
... 76 more
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:22)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:90)
... 83 more
Caused by: java.util.NoSuchElementException: Could not create a validated object, cause: ValidateObject failed
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:871)
at redis.clients.util.Pool.getResource(Pool.java:20)
... 84 more
問題調查
確定環境
發現是使用spring-data-redis通過jedis連接的redis服務端。
這個系統的代碼很久沒動，已經忘記了。先看看使用的jar版本吧。
查看應用程序使用的相關jar：
lsof -p 19377 | grep -E "jedis|pool|redis"
發現輸出的jar包含：commons-pool-1.3.jar、spring-data-redis-1.1.1.RELEASE.jar、jedis-2.1.0.jar
翻了下commons pool相關代碼
try {
_factory.activateObject(latch.getPair().value);
if(_testOnBorrow &&
!_factory.validateObject(latch.getPair().value)) {
throw new Exception("ValidateObject failed");
}
synchronized(this) {
_numInternalProcessing--;
_numActive++;
}
return latch.getPair().value;
}
catch (Throwable e) {
PoolUtils.checkRethrow(e);
// object cannot be activated or is invalid
try {
_factory.destroyObject(latch.getPair().value);
} catch (Throwable e2) {
PoolUtils.checkRethrow(e2);
// cannot destroy broken object
}
synchronized (this) {
_numInternalProcessing--;
if (!newlyCreated) {
latch.reset();
_allocationQueue.add(0, latch);
}
allocate();
}
if(newlyCreated) {
throw new NoSuchElementException("Could not create a validated object, cause: " + e.getMessage());
}
else {
continue; // keep looping
}
}
可見客戶端應該是配置了testOnBorrow，在校驗連接時失敗了。
java操作redis有多種客戶端，項目使用spring-data-redis操作redis，在spring-data-redis中也有不同的客戶端實現如jedis，lettuce等。根據錯誤日志推斷使用的redis客戶端實現為jedis。
查看JedisConnectionFactory源碼
在JedisPool中定義了校驗對象的代碼。
public boolean validateObject(final Object obj) {
if (obj instanceof Jedis) {
final Jedis jedis = (Jedis) obj;
try {
return jedis.isConnected() && jedis.ping().equals("PONG");
} catch (final Exception e) {
return false;
}
} else {
return false;
}
}
通過wireshark查看TCP包并確定問題原因
熟悉redis的同學都知道，redis客戶端發送“PING”后服務端會返回一個“PONG“作為回應，一般會作為連接的檢驗方法。
既然校驗報錯，那抓包看看請求和響應吧！
首先查看網卡編號ip a
再使用tcpdump對eth1網卡的6379端口數據抓包。
tcpdump -i eth1 port 6379 -w target.cap
最后使用wireshark對target.cap進行分析，可借助wireshark的redis插件進行分析。
根據應用錯誤日志打印的時間，查詢到此時客戶端（應用服務器）向服務端（redis服務器）發送了一個RST包。
感覺是有問題的。就往上查了下。
可以看到，箭頭位置上方客戶端發送了PING命令，箭頭位置應該返回客戶端一個PONG作為響應。而是返回了以下信息：
MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
意思是，redis服務端配置了RDB快照持久化，但當前不能進行持久化。有可能修改數據集的命令都被禁用了。（但是通過看源碼發現，除了涉及修改的命令，PING也在禁用之列，redis-3.2.9 server.c，而讀取涉及的命令應該不會受到影響）
以下代碼是redis-3.2.9 server.c中in processCommand(client *c)發生持久化異常后的處理代碼
/* Don't accept write commands if there are problems persisting on disk
* and if this is a master instance. */
if (((server.stop_writes_on_bgsave_err &&
server.saveparamslen > 0 &&
server.lastbgsave_status == C_ERR) ||
server.aof_last_write_status == C_ERR) &&
server.masterhost == NULL &&
(c->cmd->flags & CMD_WRITE ||
c->cmd->proc == pingCommand))
{
flagTransaction(c);
if (server.aof_last_write_status == C_OK)
addReply(c, shared.bgsaveerr);
else
addReplySds(c,
sdscatprintf(sdsempty(),
"-MISCONF Errors writing to the AOF file: %s\r\n",
strerror(server.aof_last_write_errno)));
return C_OK;
}
之后客戶端發送QUIT命令退出，服務器返回OK響應退出成功。
那個返回的配置錯誤信息是說在持久化RDB時出現了問題。于是到redis服務器上看了下磁盤信息和redis的日志，果然，磁盤空間不足了。
到此，問題基本查明，是由于redis所在服務器磁盤不足導致，由于是測試服務器，也沒有配置磁盤的監控。騰出空間后即可恢復。
對RST包的理解
但是我還有一個問題，那就是為什么會有一個RST包呢？如果沒有那個RST包，其實問題還不好發現，雖然按照錯誤日志的時間，挨個查找Redis數據包的信息，能夠查詢出來，但是RST無疑從一開始就吸引了我的注意，讓我能夠更加快速的定位問題。
初識RST
那現在問題來了，為什么會有RST包呢？
首先了解一下RST。（可參考TCP/IP詳解 卷1 ， 18.7 復位報文段)
歸納起來，當以下任一情況發生時，會產生RST包：
- 到不存在的端口的連接請求
- 異常終止一個連接
- 檢測半打開連接
jedis與redis的關閉機制
觀察RST之前的幾個包
使用wireshark的專家信息查看多個RST包，發現RST之前都會有QUIT,OK的交互。那看來應該是框架層面的問題。
再翻看上面GenericObjectPool的相關代碼，在borrowObject時如果發生異常，會調用destroyObject()方法，這個destroyObject是延遲到子類實現的，也就是上面說到的JedisPool。
public void destroyObject(final Object obj) throws Exception {
if (obj instanceof Jedis) {
final Jedis jedis = (Jedis) obj;
if (jedis.isConnected()) {
try {
try {
jedis.quit();
} catch (Exception e) {
}
jedis.disconnect();
} catch (Exception e) {
}
}
}
}
最終調用redis.clients.jedis.Connection的disconnect，關閉輸入輸出流。
public void disconnect() {
if (isConnected()) {
try {
inputStream.close();
outputStream.close();
if (!socket.isClosed()) {
socket.close();
}
} catch (IOException ex) {
throw new JedisConnectionException(ex);
}
}
}
這也就解釋了為什么會出現RST包：
客戶端請求QUIT，服務端返回OK。（此時客戶端在接收完quit返回后，調用了disconnect方法，導致連接斷開）緊接著服務端發起TCP揮手，發送FIN包到之前交互的客戶端51311端口，但調用完disconnect的客戶端已經斷開了和服務端的連接。客戶端只能通過發送RST，通知服務端“你發送了一個到不存在的端口的關閉請求”。
翻看新版的jedis代碼，除了將之前JedisPool中實現的代碼挪到了JedisFactory中實現，大致邏輯依然沒有改變()
// 2.10 JedisFactory
@Override
public void destroyObject(PooledObject<Jedis> pooledJedis) throws Exception {
final BinaryJedis jedis = pooledJedis.getObject();
if (jedis.isConnected()) {
try {
try {
jedis.quit();
} catch (Exception e) {
}
jedis.disconnect();
} catch (Exception e) {
}
}
}
@Override
public boolean validateObject(PooledObject<Jedis> pooledJedis) {
final BinaryJedis jedis = pooledJedis.getObject();
try {
HostAndPort hostAndPort = this.hostAndPort.get();
String connectionHost = jedis.getClient().getHost();
int connectionPort = jedis.getClient().getPort();
return hostAndPort.getHost().equals(connectionHost)
&& hostAndPort.getPort() == connectionPort && jedis.isConnected()
&& jedis.ping().equals("PONG");
} catch (final Exception e) {
return false;
}
}
而disconnect最終調用的Connection有變化。
public void disconnect() {
if (isConnected()) {
try {
outputStream.flush();
socket.close();
} catch (IOException ex) {
broken = true;
throw new JedisConnectionException(ex);
} finally {
IOUtils.closeQuietly(socket);
}
}
}
由之前的inpusStream.close()和outputStream.close()改成了outputStream.flush()。原因是jedis自定義了帶緩沖的RedisOutputStream，在socket.close前要確保緩沖內容寫到流中。
客戶端使用disconnect確實能夠快速釋放資源，在調用disconnect時關閉了客戶端端口，回收了文件句柄資源。
試想如果在quit后，服務端就已經釋放了文件句柄，關閉了socket連接，而客戶端不調用disconnect釋放資源，就會一直占用資源，在進程結束才會釋放。
下圖也進行了驗證。第一次注釋掉disconnect中關閉socket的代碼，程序sleep10秒后退出，可以看到直到進程退出時，客戶端的連接才被關閉。而第二次是恢復注釋掉的代碼，客戶端在quit后馬上就關閉了連接釋放了資源。
redis連接開啟和關閉時的系統調用
這個問題困擾了我一天，到底怎么產生的RST包？不管是客戶端還是服務端，調用close后，都應該進行正常的四次握手吧？
我反復看了redis服務端關閉客戶端連接的源碼(redis 3.2.9 networking.c#unlinkClient)。也只是調用了系統調用close(fd)，甚至為了避免干擾還新建了一個redis實例，使用strace -f -p $pid -tt -T跟蹤關閉附近的系統調用
[pid 25442] 10:29:42.299132 epoll_wait(3, {{EPOLLIN, {u32=4, u64=4}}}, 11024, 100) = 1 <0.004041>
[pid 25442] 10:29:42.303248 accept(4, {sa_family=AF_INET, sin_port=htons(52294), sin_addr=inet_addr("192.168.3.45")}, [16]) = 5 <0.000025>
[pid 25442] 10:29:42.303356 fcntl(5, F_GETFL) = 0x2 (flags O_RDWR) <0.000014>
[pid 25442] 10:29:42.303417 fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000010>
[pid 25442] 10:29:42.303456 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000012>
[pid 25442] 10:29:42.303499 epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLIN, {u32=5, u64=5}}) = 0 <0.000011>
[pid 25442] 10:29:42.303544 epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 11024, 96) = 1 <0.073370>
[pid 25442] 10:29:42.376968 read(5, "*3\r\n$3\r\nSET\r\n$3\r\nfoo\r\n$3\r\nbar\r\n", 16384) = 31 <0.000014>
[pid 25442] 10:29:42.377071 epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN|EPOLLOUT, {u32=5, u64=5}}) = 0 <0.000013>
[pid 25442] 10:29:42.377144 epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 11024, 22) = 1 <0.000017>
[pid 25442] 10:29:42.377210 write(5, "+OK\r\n", 5) = 5 <0.000034>
[pid 25442] 10:29:42.377304 epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN, {u32=5, u64=5}}) = 0 <0.000025>
[pid 25442] 10:29:42.377377 epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 11024, 22) = 1 <0.007943>
[pid 25442] 10:29:42.385376 read(5, "*2\r\n$3\r\nGET\r\n$3\r\nfoo\r\n", 16384) = 22 <0.000013>
[pid 25442] 10:29:42.385432 epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN|EPOLLOUT, {u32=5, u64=5}}) = 0 <0.000011>
[pid 25442] 10:29:42.385477 epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 11024, 14) = 1 <0.000010>
[pid 25442] 10:29:42.385518 write(5, "$3\r\nbar\r\n", 9) = 9 <0.000019>
[pid 25442] 10:29:42.385567 epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN, {u32=5, u64=5}}) = 0 <0.000011>
[pid 25442] 10:29:42.385617 epoll_wait(3, {}, 11024, 14) = 0 <0.014075>
[pid 25442] 10:29:42.399742 epoll_wait(3, {}, 11024, 100) = 0 <0.100126>
[pid 25442] 10:29:42.499930 epoll_wait(3, {}, 11024, 100) = 0 <0.100126>
[pid 25442] 10:29:42.600115 epoll_wait(3, {}, 11024, 100) = 0 <0.100071>
[pid 25442] 10:29:42.700276 epoll_wait(3, {}, 11024, 100) = 0 <0.100131>
[pid 25442] 10:29:42.800482 epoll_wait(3, {}, 11024, 100) = 0 <0.100129>
[pid 25442] 10:29:42.900687 epoll_wait(3, {}, 11024, 100) = 0 <0.100141>
[pid 25442] 10:29:43.000895 epoll_wait(3, {}, 11024, 100) = 0 <0.100132>
[pid 25442] 10:29:43.101095 epoll_wait(3, {}, 11024, 100) = 0 <0.100131>
[pid 25442] 10:29:43.201305 epoll_wait(3, {}, 11024, 100) = 0 <0.100134>
[pid 25442] 10:29:43.301521 epoll_wait(3, {}, 11024, 100) = 0 <0.100136>
[pid 25442] 10:29:43.401725 epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 11024, 100) = 1 <0.003552>
[pid 25442] 10:29:43.405350 read(5, "*2\r\n$3\r\nGET\r\n$3\r\nfoo\r\n", 16384) = 22 <0.000016>
[pid 25442] 10:29:43.405425 epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN|EPOLLOUT, {u32=5, u64=5}}) = 0 <0.000011>
[pid 25442] 10:29:43.405477 epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 11024, 96) = 1 <0.000014>
[pid 25442] 10:29:43.405531 write(5, "$3\r\nbar\r\n", 9) = 9 <0.000022>
[pid 25442] 10:29:43.405601 epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN, {u32=5, u64=5}}) = 0 <0.000011>
[pid 25442] 10:29:43.405660 epoll_wait(3, {}, 11024, 96) = 0 <0.096129>
[pid 25442] 10:29:43.501877 epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 11024, 100) = 1 <0.003474>
[pid 25442] 10:29:43.505429 read(5, "*1\r\n$4\r\nQUIT\r\n", 16384) = 14 <0.000018>
[pid 25442] 10:29:43.505514 epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN|EPOLLOUT, {u32=5, u64=5}}) = 0 <0.000015>
[pid 25442] 10:29:43.505578 epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 11024, 96) = 1 <0.000012>
[pid 25442] 10:29:43.505623 write(5, "+OK\r\n", 5) = 5 <0.000028>
[pid 25442] 10:29:43.505693 epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN, {u32=5, u64=5}}) = 0 <0.000016>
[pid 25442] 10:29:43.505764 epoll_ctl(3, EPOLL_CTL_DEL, 5, {0, {u32=5, u64=5}}) = 0 <0.000016>
[pid 25442] 10:29:43.505830 close(5) = 0 <0.000111>
[pid 25442] 10:29:43.505992 epoll_wait(3, {}, 11024, 96) = 0 <0.096134>
java客戶端junit測試代碼(根據jedis測試用例JedisPoolTest#checkConnections修改)：
JedisPool pool = new JedisPool(new JedisPoolConfig(), hnp.getHost(), hnp.getPort(), 2000);
Jedis jedis = pool.getResource();
jedis.set("foo", "bar");
assertEquals("bar", jedis.get("foo"));
pool.returnResource(jedis);
try {
Thread.sleep(1*1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("hello");
jedis.get("foo");
pool.destroy();
assertTrue(pool.isClosed());
觀察服務端系統調用，
setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
...
close(5) = 0
在socket連接時只設置了TCP_NODELAY，禁用了Nagle算法。
jedis客戶端的socket設置
正在無解之際，突然想到是不是redis客戶端設置了一些參數呢？
終于，在jedis控制連接的redis.clients.jedisConnection類中，找到了連接時對socket的設置：
public void connect() {
if (!isConnected()) {
try {
socket = new Socket();
// ->@wjw_add
socket.setReuseAddress(true);
socket.setKeepAlive(true); // Will monitor the TCP connection is
// valid
socket.setTcpNoDelay(true); // Socket buffer Whetherclosed, to
// ensure timely delivery of data
socket.setSoLinger(true, 0); // Control calls close () method,
// the underlying socket is closed
// immediately
// <-@wjw_add
socket.connect(new InetSocketAddress(host, port), connectionTimeout);
socket.setSoTimeout(soTimeout);
if (ssl) {
if (null == sslSocketFactory) {
sslSocketFactory = (SSLSocketFactory)SSLSocketFactory.getDefault();
}
socket = (SSLSocket) sslSocketFactory.createSocket(socket, host, port, true);
if (null != sslParameters) {
((SSLSocket) socket).setSSLParameters(sslParameters);
}
if ((null != hostnameVerifier) &&
(!hostnameVerifier.verify(host, ((SSLSocket) socket).getSession()))) {
String message = String.format(
"The connection to '%s' failed ssl/tls hostname verification.", host);
throw new JedisConnectionException(message);
}
}
outputStream = new RedisOutputStream(socket.getOutputStream());
inputStream = new RedisInputStream(socket.getInputStream());
} catch (IOException ex) {
broken = true;
throw new JedisConnectionException("Failed connecting to host "
+ host + ":" + port, ex);
}
}
}
這個socket.setSoLinger(true, 0);引起了我的注意。
根據SCTP rfc SO_LINGER的解釋
If the l_linger value is set to 0, calling close() is the same as the ABORT primitive.
繼續看SCTP_ABORT:
SCTP_ABORT: Setting this flag causes the specified association
to abort by sending an ABORT message to the peer. The ABORT
chunk will contain an error cause of 'User Initiated Abort'
with cause code 12. The cause-specific information of this
error cause is provided in msg_iov.
不太明白，看下TCP中對Abort的解釋吧
TCP rfc對Abort的解釋：
This command causes all pending SENDs and RECEIVES to be
aborted, the TCB to be removed, and a special RESET message to
be sent to the TCP on the other side of the connection.
Depending on the implementation, users may receive abort
indications for each outstanding SEND or RECEIVE, or may simply
receive an ABORT-acknowledgment.
注：TCB是一個抽象的控制塊(Transmission Control Block)
Socket選項SO_LINGER用于強制中斷
到此才算明白，由于jedis客戶端在連接時，設置了socket.setSoLinger(true, 0);，這樣在關閉連接時就等同與TCP的Abort，也就是忽略所有正在發送和接收的數據，直接向對方發送一個RESET消息。這也是為什么jedis要在socket.close()前flush緩沖，以確保在途數據不會丟失。
我去掉了客戶端對SO_LINGER的設置，終于又看到了正常的TCP揮手。
還想深入的同學，可以閱讀linux源碼net/ipv4/tcp.c。我大概看了下，代碼邏輯很明確（linux內核版本有區別）如果設置了SO_LINGER，在close時，會直接調用tcp_disconnect發送RST數據包，而不再做常規的四次揮手流程。雖然我覺得這樣做不太優雅，更優雅的做法可能是socket.setSoLinger(true, timeout)設置一個超時閥值。
在這個github jedis issue Improving socket performance中描述了加入以下四項設置用于提升性能。
socket.setReuseAddress(true);
socket.setKeepAlive(true);
socket.setTcpNoDelay(true);
socket.setSoLinger(true,0);
在issue下加了個comment詢問了下，有消息了再更新吧。
總結
此次應用程序中Jedis連接池不能獲取redis連接的問題，原因是redis服務器磁盤空間滿，導致不能保存快照(rdb snapshot)。應用程序中在testOnBorrow為true的情況下，使用redisPING PONG命令測試redis連接是否有效時，收到了MISCONF Redis is configured to save RDB snapshots的響應，而非正常的PONG。這就導致jedis判斷連接無效，強制斷開了連接。
之后對TCP中RST flag做了淺嘗輒止的分析。當設置了socket.setSoLinger(true, 0)后，關閉此socket將清空數據并向對方發送RST消息。
可以深入的地方還有不少，自己關于網絡編程的知識也有待加強。準備補充下相關知識，再結合一些優秀的開源項目如redis、nginx深入了解下。
參考
- Jedis源碼 https://github.com/xetorthio/jedis
- Commons-pool源碼 https://github.com/apache/commons-pool
- Spring-data-redis源碼 https://github.com/spring-projects/spring-data-redis
- redis-wireshark源碼 https://github.com/jzwinck/redis-wireshark
- Redis源碼 https://github.com/antirez/redis
- TCP/IP詳解在線電子書 http://www.52im.net/topic-tcpipvol1.html
- SCTP rfc - https://tools.ietf.org/html/rfc6458
- TCP rfc - https://tools.ietf.org/html/rfc793
- 幾種TCP連接中出現RST的情況
- setsockopt()--Set Socket Options
- StackOverflow What is AF_INET, and why do I need it?
- Socket選項系列之SO_LINGER(《深入剖析Nginx》作者) - http://www.lenky.info/archives/2013/02/2220
