用于快速向redis插入大量數據。官方文檔
官方理由大概如此:
Using a normal Redis client to perform mass insertion is not a good idea for a few reasons: the naive approach of sending one command after the other is slow because you have to pay for the round trip time for every command. It is possible to use pipelining, but for mass insertion of many records you need to write new commands while you read replies at the same time to make sure you are inserting as fast as possible.
Only a small percentage of clients support non-blocking I/O, and not all the clients are able to parse the replies in an efficient way in order to maximize throughput. For all this reasons the preferred way to mass import data into Redis is to generate a text file containing the Redis protocol, in raw format, in order to call the commands needed to insert the required data.
由于redis命令之間存在往返時延,即使并發客戶端寫入redis,效果仍然不能讓人滿意。因此redis2.6發布支持pipe line模式,支持快速大量插入。步驟如下:
1.構建redis命令文本,如redis_command.txt
拿到原始的數據,通過業務語言構建此文件不成問題,生成如下格式的命令文本
SET Key0 Value0
SET Key1 Value1
...
SET KeyN ValueN
2.生成Redis Protocal的文本
github上找了個shell版本的,transfer.sh
#!/bin/bash
while read CMD; do
# each command begins with *{number arguments in command}\r\n
XS=($CMD); printf "*${#XS[@]}\r\n"
# for each argument, we append ${length}\r\n{argument}\r\n
for X in $CMD; do printf "\$${#X}\r\n$X\r\n"; done
done < redis_commands.txt
執行命令:
sh transfer.sh > redis_protocal.txt
3.調用redis pipe,完成批量插入數據
cat redis_protocal.txt | redis-cli -h host -p port --pipe