原創(chuàng)文章,歡迎轉(zhuǎn)載。轉(zhuǎn)載請注明:轉(zhuǎn)載自IT人故事會,謝謝!
原文鏈接地址:『互聯(lián)網(wǎng)架構(gòu)』軟件架構(gòu)-netty粘包分包編碼解碼(57)
一般直接接觸RPC框架的時候內(nèi)部都做了對于粘包分包的解決方案,咱們來一起了解下這方便的含義,包括編碼解碼這塊。
源碼:https://github.com/limingios/netFuture/tree/master/源碼/『互聯(lián)網(wǎng)架構(gòu)』軟件架構(gòu)-io與nio線程模型reactor模型(上)(53)/nio
(一)粘包分包概念
- 粘包
TCP
由于TCP協(xié)議本身的機制(面向連接的可靠地協(xié)議-三次握手機制)客戶端與服務(wù)器會維持一個連接(Channel),數(shù)據(jù)在連接不斷開的情況下,可以持續(xù)不斷地將多個數(shù)據(jù)包發(fā)往服務(wù)器,但是如果發(fā)送的網(wǎng)絡(luò)數(shù)據(jù)包太小,那么他本身會啟用Nagle算法(可配置是否啟用)對較小的數(shù)據(jù)包進行合并(基于此,TCP的網(wǎng)絡(luò)延遲要UDP的高些)然后再發(fā)送(超時或者包大小足夠)。那么這樣的話,服務(wù)器在接收到消息(數(shù)據(jù)流)的時候就無法區(qū)分哪些數(shù)據(jù)包是客戶端自己分開發(fā)送的,這樣產(chǎn)生了粘包;服務(wù)器在接收到數(shù)據(jù)庫后,放到緩沖區(qū)中,如果消息沒有被及時從緩存區(qū)取走,下次在取數(shù)據(jù)的時候可能就會出現(xiàn)一次取出多個數(shù)據(jù)包的情況,造成粘包現(xiàn)象(確切來講,對于基于TCP協(xié)議的應(yīng)用,不應(yīng)用包來描述,而應(yīng) 用 流來描述),個人認為服務(wù)器接收端產(chǎn)生的粘包應(yīng)該與linux內(nèi)核處理socket的方式 select輪詢機制的線性掃描頻度無關(guān)。
UDP
本身作為無連接的不可靠的傳輸協(xié)議(適合頻繁發(fā)送較小的數(shù)據(jù)包),他不會對數(shù)據(jù)包進行合并發(fā)送(也就沒有Nagle算法之說了),他直接是一端發(fā)送什么數(shù)據(jù),直接就發(fā)出去了,既然他不會對數(shù)據(jù)合并,每一個數(shù)據(jù)包都是完整的(數(shù)據(jù)+UDP頭+IP頭等等發(fā)一次數(shù)據(jù)封裝一次)也就沒有粘包一說了。
- 分包
可能是IP分片傳輸導(dǎo)致的,也可能是傳輸過程中丟失部分包導(dǎo)致出現(xiàn)的半包,還有可能就是一個包可能被分成了兩次傳輸,在取數(shù)據(jù)的時候,先取到了一部分(還可能與接收的緩沖區(qū)大小有關(guān)系),總之就是一個數(shù)據(jù)包被分成了多次接收。
- TCP當(dāng)中,只有流的概念,沒有包的概念(根本原因)
簡單的概括
(1)粘包:
1.服務(wù)端
原因收到的數(shù)據(jù)放在系統(tǒng)接收緩沖區(qū),用戶進程從該緩沖區(qū)取數(shù)據(jù)
2.客戶端
原因TCP為提高傳輸效率,要收集到足夠多的數(shù)據(jù)后才發(fā)送一包數(shù)據(jù)
(2).分包:
1.應(yīng)用程序?qū)懭氲淖止?jié)大小大于套接字發(fā)送緩沖區(qū)的大小
2.進行mss(最大報文長度)大小的TCP分段,當(dāng)TCP報文長度-TCP頭部長度>MSS
3.以太網(wǎng)幀的payload(凈荷)大于MTU(1500字節(jié))進行ip分片
(二)Netty粘包分包現(xiàn)象演示
源碼:pack目錄下的error
Server.java
package com.dig8.pack.error;
import org.jboss.netty.bootstrap.ServerBootstrap;
import org.jboss.netty.channel.ChannelPipeline;
import org.jboss.netty.channel.ChannelPipelineFactory;
import org.jboss.netty.channel.Channels;
import org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory;
import java.net.InetSocketAddress;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
/**
* netty服務(wù)端
*
* @author idig8.com
*/
public class Server {
public static void main(String[] args) {
// 服務(wù)類
ServerBootstrap bootstrap = new ServerBootstrap();
// boss線程,主要監(jiān)聽端口和獲取worker線程及分配socketChannel給worker線程
ExecutorService boss = Executors.newCachedThreadPool();
// worker線程負責(zé)數(shù)據(jù)讀寫
ExecutorService worker = Executors.newCachedThreadPool();
// 設(shè)置niosocket工廠
bootstrap.setFactory(new NioServerSocketChannelFactory(boss, worker));
// 設(shè)置管道的工廠
bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
@Override
public ChannelPipeline getPipeline() throws Exception {
ChannelPipeline pipeline = Channels.pipeline();
// 管道過濾器
pipeline.addLast("myHandler", new ServerHandler());
return pipeline;
}
});
// 服務(wù)類綁定端口
bootstrap.bind(new InetSocketAddress(8888));
}
}
ServerHandler.java
package com.dig8.pack.error;
import org.jboss.netty.buffer.ChannelBuffer;
import org.jboss.netty.channel.ChannelHandlerContext;
import org.jboss.netty.channel.MessageEvent;
import org.jboss.netty.channel.SimpleChannelHandler;
/**
* @author idig8.com
*/
public class ServerHandler extends SimpleChannelHandler{
@Override
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) throws Exception {
ChannelBuffer buffer = (ChannelBuffer) e.getMessage();
byte[] bs = buffer.array();
System.out.println("server receive data: " +new String(bs));
}
}
Client.java
package com.dig8.pack.error;
import org.jboss.netty.bootstrap.ClientBootstrap;
import org.jboss.netty.channel.ChannelPipeline;
import org.jboss.netty.channel.ChannelPipelineFactory;
import org.jboss.netty.channel.Channels;
import org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory;
import org.jboss.netty.handler.codec.string.StringEncoder;
import java.net.InetSocketAddress;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
/**
* 客戶端
*
* @author idig8.com
*/
public class Client {
public static void main(String[] args) throws Exception {
//服務(wù)類
ClientBootstrap bootstrap = new ClientBootstrap();
//線程池
ExecutorService boss = Executors.newCachedThreadPool();
ExecutorService worker = Executors.newCachedThreadPool();
//socket工廠
bootstrap.setFactory(new NioClientSocketChannelFactory(boss, worker));
//管道工廠
bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
@Override
public ChannelPipeline getPipeline() throws Exception {
ChannelPipeline pipeline = Channels.pipeline();
pipeline.addLast("1", new StringEncoder());
pipeline.addLast("2", new ClientHandler());
return pipeline;
}
});
//連接服務(wù)端
bootstrap.connect(new InetSocketAddress("127.0.0.1", 8888)).sync();
}
}
ClientHandler.java
package com.dig8.pack.error;
import org.jboss.netty.buffer.ChannelBuffer;
import org.jboss.netty.buffer.ChannelBuffers;
import org.jboss.netty.channel.*;
/**
* 客戶端消息處理類
* @author idig8.com
*/
public class ClientHandler extends SimpleChannelHandler {
// 包頭
private static final int HEAD_FLAG = -32323231;
@Override
public void channelConnected(ChannelHandlerContext ctx, ChannelStateEvent e) throws Exception {
Channel channel = ctx.getChannel();
String msg = "Hello,idig8.com";
for (int i = 0; i < 1000; i++) {
channel.write(msg);
}
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) throws Exception {
}
}
運行后出現(xiàn)粘包和分包現(xiàn)象
(三)粘包分包問題解決思路
服務(wù)端和客戶端約定好穩(wěn)定的數(shù)據(jù)包結(jié)構(gòu)
1.客戶端根據(jù)約定的數(shù)據(jù)包結(jié)構(gòu)發(fā)送數(shù)據(jù)
2.服務(wù)端根據(jù)約定的數(shù)據(jù)包結(jié)構(gòu)來讀取數(shù)據(jù)
通過MyDecoder集成FrameDecoder的方式來
源碼:pack目錄下的custom
Server.java
package com.dig8.pack.custom;
import java.net.InetSocketAddress;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import org.jboss.netty.bootstrap.ServerBootstrap;
import org.jboss.netty.channel.ChannelPipeline;
import org.jboss.netty.channel.ChannelPipelineFactory;
import org.jboss.netty.channel.Channels;
import org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory;
/**
* netty服務(wù)端
*
* @author idig8.com
*/
public class Server {
public static void main(String[] args) {
// 服務(wù)類
ServerBootstrap bootstrap = new ServerBootstrap();
// boss線程,主要監(jiān)聽端口和獲取worker線程及分配socketChannel給worker線程
ExecutorService boss = Executors.newCachedThreadPool();
// worker線程負責(zé)數(shù)據(jù)讀寫
ExecutorService worker = Executors.newCachedThreadPool();
// 設(shè)置niosocket工廠
bootstrap.setFactory(new NioServerSocketChannelFactory(boss, worker));
// 設(shè)置管道的工廠
bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
@Override
public ChannelPipeline getPipeline() throws Exception {
ChannelPipeline pipeline = Channels.pipeline();
// 管道過濾器
pipeline.addLast("myDecoder", new MyDecoder());
pipeline.addLast("myHandler", new ServerHandler());
return pipeline;
}
});
// 服務(wù)類綁定端口
bootstrap.bind(new InetSocketAddress(7778));
}
}
ServerHandler.java
package com.dig8.pack.custom;
import org.jboss.netty.buffer.ChannelBuffer;
import org.jboss.netty.channel.ChannelHandlerContext;
import org.jboss.netty.channel.MessageEvent;
import org.jboss.netty.channel.SimpleChannelHandler;
/**
* @author idig8.com
*/
public class ServerHandler extends SimpleChannelHandler{
@Override
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) throws Exception {
/*ChannelBuffer buffer = (ChannelBuffer) e.getMessage();
byte[] bs = buffer.array();*/
System.out.println("server receive data: " + e.getMessage());
}
}
Client.java
package com.dig8.pack.custom;
import java.net.InetSocketAddress;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import org.jboss.netty.bootstrap.ClientBootstrap;
import org.jboss.netty.channel.ChannelPipeline;
import org.jboss.netty.channel.ChannelPipelineFactory;
import org.jboss.netty.channel.Channels;
import org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory;
import org.jboss.netty.handler.codec.string.StringEncoder;
/**
* 客戶端
*
* @author idig8.com
*/
public class Client {
public static void main(String[] args) throws Exception {
//服務(wù)類
ClientBootstrap bootstrap = new ClientBootstrap();
//線程池
ExecutorService boss = Executors.newCachedThreadPool();
ExecutorService worker = Executors.newCachedThreadPool();
//socket工廠
bootstrap.setFactory(new NioClientSocketChannelFactory(boss, worker));
//管道工廠
bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
@Override
public ChannelPipeline getPipeline() throws Exception {
ChannelPipeline pipeline = Channels.pipeline();
pipeline.addLast("1", new StringEncoder());
pipeline.addLast("2", new ClientHandler());
return pipeline;
}
});
//連接服務(wù)端
bootstrap.connect(new InetSocketAddress("127.0.0.1", 7778)).sync();
}
}
ClientHandler.java
package com.dig8.pack.custom;
import org.jboss.netty.buffer.ChannelBuffer;
import org.jboss.netty.buffer.ChannelBuffers;
import org.jboss.netty.channel.Channel;
import org.jboss.netty.channel.ChannelHandlerContext;
import org.jboss.netty.channel.ChannelStateEvent;
import org.jboss.netty.channel.ExceptionEvent;
import org.jboss.netty.channel.SimpleChannelHandler;
/**
* 客戶端消息處理類
* @author idig8.com
*/
public class ClientHandler extends SimpleChannelHandler {
// 包頭
private static final int HEAD_FLAG = -32323231;
@Override
public void channelConnected(ChannelHandlerContext ctx, ChannelStateEvent e) throws Exception {
Channel channel = ctx.getChannel();
String msg = "Hello,idig8.com 通過定義包頭+長度+數(shù)據(jù) 防止粘包和分包";
byte[] bytes = msg.getBytes();
// 定義數(shù)據(jù)包 ,結(jié)構(gòu)為:包頭 + 長度 + 數(shù)據(jù)
ChannelBuffer buffer = ChannelBuffers.dynamicBuffer();
// 1.寫包頭
buffer.writeInt(HEAD_FLAG);// 4字節(jié)
// 2.寫長度
buffer.writeInt(bytes.length);// 4字節(jié)
// 3.寫數(shù)據(jù)本身
buffer.writeBytes(bytes);
for (int i = 0; i < 1000; i++) {
channel.write(buffer);
}
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) throws Exception {
}
}
MyDecoder.java
/**
*
*/
package com.dig8.pack.custom;
import org.jboss.netty.buffer.ChannelBuffer;
import org.jboss.netty.channel.Channel;
import org.jboss.netty.channel.ChannelHandlerContext;
import org.jboss.netty.handler.codec.frame.FrameDecoder;
/**
* @author idig8.com
*/
public class MyDecoder extends FrameDecoder{
// 包頭
private static final int HEAD_FLAG = -32323231;
// 數(shù)據(jù)包基本長度
private final static int BASE_LENGTH = 4 + 4;
@Override
protected Object decode(ChannelHandlerContext ctx, Channel channel, ChannelBuffer buffer) throws Exception {
// 收到數(shù)據(jù)之后,先判斷buffer中可讀的數(shù)據(jù)長度是否大于數(shù)據(jù)包的基本長度
if(buffer.readableBytes() > BASE_LENGTH){
// 防止socket攻擊:
if(buffer.readableBytes() > 4096 * 2){ // 4k
System.out.println("socket 攻擊了");
buffer.skipBytes(buffer.readableBytes());
}
// 記錄包頭開始的位置
int headIndex;
while(true){
headIndex = buffer.readerIndex();
buffer.markReaderIndex();
// 代碼很關(guān)鍵
if(buffer.readableBytes() < 4){// 包頭的長度
buffer.readerIndex(headIndex);
return null;
}
// 此時說明包頭的長度是足夠的
// 正好讀取的是包頭
if(buffer.readInt() == HEAD_FLAG ){
break;
}
// [1,2,3,4] 1 1 1 1 1
// 如果不是包頭,需要略過一個字節(jié),在略過之前,需要還原讀指針位置
buffer.resetReaderIndex();
buffer.readByte();// 略過一個字節(jié)
if(buffer.readableBytes() < BASE_LENGTH){
return null;
}
}
// 此時說明有數(shù)據(jù)包到來
// 做標記(記住當(dāng)前讀指針的位置)
// buffer.markReaderIndex();
// 1.讀長度
int dataLength = buffer.readInt();
if(buffer.readableBytes() < dataLength){
// 說明數(shù)據(jù)本身的長度還不夠, 肯定要繼續(xù)等待后面的數(shù)據(jù)到來
// 還原讀指針的位置
buffer.readerIndex(headIndex);
return null;
}
// 此時說明數(shù)據(jù)包已經(jīng)位置
// 2.讀數(shù)據(jù)本身
byte[] dst = new byte[dataLength];
buffer.readBytes(dst);
// 繼續(xù)傳遞下去
// ?
// 如果此時buffer中的數(shù)據(jù)還沒有讀完,那么剩下的數(shù)據(jù)怎么辦?
return new String(dst);
}
// return null 表示此時的數(shù)據(jù)包不完整,需要繼續(xù)等待下一個數(shù)據(jù)包的到來 ?
return null;
}
}
(三)Netty自帶粘包分包解決方案
消息定長
1.FixedLengthFrameDecoder
行分隔符
2.LineBasedFrameDecoder
自定義特殊符號進行分割
3.DelimiterBasedFrameDecoder
源碼:pack目錄下的nettysolution
Server.java
package com.dig8.pack.nettysolution;
import java.net.InetSocketAddress;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import org.jboss.netty.bootstrap.ServerBootstrap;
import org.jboss.netty.buffer.ChannelBuffers;
import org.jboss.netty.channel.ChannelPipeline;
import org.jboss.netty.channel.ChannelPipelineFactory;
import org.jboss.netty.channel.Channels;
import org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory;
import org.jboss.netty.handler.codec.frame.DelimiterBasedFrameDecoder;
import org.jboss.netty.handler.codec.frame.FixedLengthFrameDecoder;
import org.jboss.netty.handler.codec.frame.LineBasedFrameDecoder;
import org.jboss.netty.handler.codec.string.StringDecoder;
/**
* 服務(wù)端
* @author idig8.com
*/
public class Server {
public static void main(String[] args) throws Exception {
// 服務(wù)類
ServerBootstrap bootstrap = new ServerBootstrap();
// boss線程,主要監(jiān)聽端口和獲取worker線程及分配socketChannel給worker線程
ExecutorService boss = Executors.newCachedThreadPool();
// worker線程負責(zé)數(shù)據(jù)讀寫
ExecutorService worker = Executors.newCachedThreadPool();
// 設(shè)置niosocket工廠
bootstrap.setFactory(new NioServerSocketChannelFactory(boss, worker));
// 設(shè)置管道的工廠
bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
@Override
public ChannelPipeline getPipeline() throws Exception {
ChannelPipeline pipeline = Channels.pipeline();
// 管道過濾器
// 方案1:消息定長
//pipeline.addLast("fixedLength", new FixedLengthFrameDecoder(18));
// 方案2:行分隔符
//pipeline.addLast("fixedLength", new LineBasedFrameDecoder(1024));
// 方案3:自定義特殊符號進行分割
pipeline.addLast("delimiter", new DelimiterBasedFrameDecoder(1024,
ChannelBuffers.copiedBuffer("#@#".getBytes())));
pipeline.addLast("1",new StringDecoder());
pipeline.addLast("2",new ServerMessageHandler());
return pipeline;
}
});
// 服務(wù)類綁定端口
bootstrap.bind(new InetSocketAddress(7777));
System.out.println("服務(wù)端啟動...");
}
}
ServerMessageHandler.java
package com.dig8.pack.nettysolution;
import org.jboss.netty.channel.ChannelHandlerContext;
import org.jboss.netty.channel.ExceptionEvent;
import org.jboss.netty.channel.MessageEvent;
import org.jboss.netty.channel.SimpleChannelHandler;
/**
* 服務(wù)端消息處理類
* @author idig8.com
*/
public class ServerMessageHandler extends SimpleChannelHandler {
/**
* 接收消息
*/
@Override
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) throws Exception {
System.out.println("receive request: " + e.getMessage());
}
/**
* 異常處理
*/
@Override
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) throws Exception {
}
}
Client.java
package com.dig8.pack.nettysolution;
import java.net.InetSocketAddress;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import org.jboss.netty.bootstrap.ClientBootstrap;
import org.jboss.netty.buffer.ChannelBuffer;
import org.jboss.netty.buffer.ChannelBuffers;
import org.jboss.netty.channel.ChannelFuture;
import org.jboss.netty.channel.ChannelPipeline;
import org.jboss.netty.channel.ChannelPipelineFactory;
import org.jboss.netty.channel.Channels;
import org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory;
import org.jboss.netty.handler.codec.frame.DelimiterBasedFrameDecoder;
import org.jboss.netty.handler.codec.frame.FixedLengthFrameDecoder;
import org.jboss.netty.handler.codec.frame.LineBasedFrameDecoder;
import org.jboss.netty.handler.codec.string.StringEncoder;
/**
* 客戶端
*
* @author idig8.com
*/
public class Client {
public static void main(String[] args) throws Exception {
//服務(wù)類
ClientBootstrap bootstrap = new ClientBootstrap();
//線程池
ExecutorService boss = Executors.newCachedThreadPool();
ExecutorService worker = Executors.newCachedThreadPool();
//socket工廠
bootstrap.setFactory(new NioClientSocketChannelFactory(boss, worker));
//管道工廠
bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
@Override
public ChannelPipeline getPipeline() throws Exception {
ChannelPipeline pipeline = Channels.pipeline();
// 方案1:消息定長
//pipeline.addLast("fixedLength", new FixedLengthFrameDecoder(18));
// 方案2:行分隔符
//pipeline.addLast("fixedLength", new LineBasedFrameDecoder(1024));
// 方案3:自定義特殊符號進行分割
pipeline.addLast("delimiter", new DelimiterBasedFrameDecoder(1024,
ChannelBuffers.copiedBuffer("#@#".getBytes())));
pipeline.addLast("1",new StringEncoder());
pipeline.addLast("2", new ClientMessageHandler());
return pipeline;
}
});
//連接服務(wù)端
@SuppressWarnings("unused")
ChannelFuture connect = bootstrap.connect(new InetSocketAddress("127.0.0.1", 7777)).sync();
// Channel channel = connect.getChannel();
// System.out.println("client start");
// Scanner scanner = new Scanner(System.in);
// while(true){
// System.out.println("請輸入:");
// channel.write(scanner.next());
// }
}
}
ClientMessageHandler.java
package com.dig8.pack.nettysolution;
import org.jboss.netty.channel.Channel;
import org.jboss.netty.channel.ChannelHandlerContext;
import org.jboss.netty.channel.ChannelStateEvent;
import org.jboss.netty.channel.ExceptionEvent;
import org.jboss.netty.channel.MessageEvent;
import org.jboss.netty.channel.SimpleChannelHandler;
/**
* 客戶端消息接受處理類
* @author idig8.com
*/
public class ClientMessageHandler extends SimpleChannelHandler {
/**
* 接收消息
*/
@Override
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) throws Exception {
System.out.println("server response : " + e.getMessage());
}
/**
* 新連接
*/
@Override
public void channelConnected(ChannelHandlerContext ctx, ChannelStateEvent e) throws Exception {
Channel channel = ctx.getChannel();
String separator = "#@#";//System.getProperty("line.separator");// 系統(tǒng)換行符
String msg = "idig8.com send cmd";
for (int i = 0; i < 1000; i++) {
channel.write(msg + i + separator);
}
}
/**
* 異常處理
*/
@Override
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) throws Exception {
}
}
PS:基本上netty針對tcp 分包粘包已經(jīng)說完了,確實有了netty真的很方便比傳統(tǒng)的socket方便很多。下次說說http 協(xié)議實現(xiàn)。