本文以常見的NioEventLoop為切入點分析Netty的EventLoop,NioEventLoop的類層次結構如下圖所示,下面將按照類層次結構自底向上依次分析。
AbstractEventExecutor類
AbstractEventExecutor類實現了EventExecutor接口,比較重要的是parent成員變量,用于引用該EventExecutor所屬的EventExecutorGroup,以schedule開頭的調度方法都拋出了UnsupportedOperationException,子類需要覆蓋這些方法。
public abstract class AbstractEventExecutor extends AbstractExecutorService implements EventExecutor {
private final EventExecutorGroup parent;
// 省略一些代碼
protected AbstractEventExecutor() {
this(null);
}
protected AbstractEventExecutor(EventExecutorGroup parent) {
this.parent = parent;
}
@Override
public EventExecutorGroup parent() {
return parent;
}
@Override
public EventExecutor next() {
return this;
}
@Override
public boolean inEventLoop() {
return inEventLoop(Thread.currentThread());
}
@Override
public ScheduledFuture<?> schedule(Runnable command, long delay,
TimeUnit unit) {
throw new UnsupportedOperationException();
}
// 省略一些代碼
}
AbstractScheduledEventExecutor類
AbstractScheduledEventExecutor類繼承了AbstractEventExecutor類,實現了調度方法以支持任務調度。
SingleThreadEventExecutor類
SingleThreadEventExecutor類繼承了AbstractScheduledEventExecutor類,從類名可以看出該EventExecutor是單線程的。
成員變量和構造函數
該類重要的成員變量和構造函數如下所示:
public abstract class SingleThreadEventExecutor extends AbstractScheduledEventExecutor implements OrderedEventExecutor {
static final int DEFAULT_MAX_PENDING_EXECUTOR_TASKS = Math.max(16,
SystemPropertyUtil.getInt("io.netty.eventexecutor.maxPendingTasks", Integer.MAX_VALUE));
private final Queue<Runnable> taskQueue;
private volatile Thread thread;
private volatile ThreadProperties threadProperties;
private final Executor executor;
private volatile boolean interrupted;
// 省略一些代碼
protected SingleThreadEventExecutor(
EventExecutorGroup parent, ThreadFactory threadFactory, boolean addTaskWakesUp) {
this(parent, new ThreadPerTaskExecutor(threadFactory), addTaskWakesUp);
}
protected SingleThreadEventExecutor(
EventExecutorGroup parent, ThreadFactory threadFactory,
boolean addTaskWakesUp, int maxPendingTasks, RejectedExecutionHandler rejectedHandler) {
this(parent, new ThreadPerTaskExecutor(threadFactory), addTaskWakesUp, maxPendingTasks, rejectedHandler);
}
protected SingleThreadEventExecutor(EventExecutorGroup parent, Executor executor, boolean addTaskWakesUp) {
this(parent, executor, addTaskWakesUp, DEFAULT_MAX_PENDING_EXECUTOR_TASKS, RejectedExecutionHandlers.reject());
}
protected SingleThreadEventExecutor(EventExecutorGroup parent, Executor executor,
boolean addTaskWakesUp, int maxPendingTasks,
RejectedExecutionHandler rejectedHandler) {
super(parent);
this.addTaskWakesUp = addTaskWakesUp;
this.maxPendingTasks = Math.max(16, maxPendingTasks);
this.executor = ObjectUtil.checkNotNull(executor, "executor");
taskQueue = newTaskQueue(this.maxPendingTasks);
rejectedExecutionHandler = ObjectUtil.checkNotNull(rejectedHandler, "rejectedHandler");
}
// 省略一些代碼
}
- taskQueue是任務隊列,用來存放需要調度執行的任務;
- thread用來引用支撐該EventExecutor的線程,用來處理I/O事件和執行任務,叫支撐線程或者I/O線程均可;
- executor用來引用線程池,thread所引用的線程即來自這里。
inEventLoop方法
其父類AbstractEventExecutor類中的inEventLoop()實現調用了inEventLoop(Thread.currentThread()),以下是其實現:
@Override
public boolean inEventLoop(Thread thread) {
return thread == this.thread;
}
可以看到是用該EventExecutor所綁定的線程與參數去做比較,所以inEventLoop()的語義較為明晰:如果當前運行的線程是EventExecutor的支撐線程則返回true,否則返回false。這一方法在DefaultChannelPipeline和AbstractChannelHandlerContext類中意義重大,如《Netty實戰》7.4.1節所述:
Netty線程模型的卓越性能取決于對于當前執行線程的身份的確定,也就是說,確定它是否是分配給當前Channel以及它的EventLoop的那一個線程。
如果(當前)調用線程正是支撐EventLoop的線程,那么所提交的代碼塊將會被(直接)執行。否則,EventLoop將調度該任務以便稍后執行,并將它放入到內部隊列中。當EventLoop下次處理它的事件時,它會執行隊列中的那些任務。這也就解釋了任何的線程是如何與Channel直接交互而無需在ChannelHandler中進行額外同步的。
任務執行與線程綁定
execute方法實現了Java并發包Executor的接口方法,用來執行任務(如通道注冊等):
- addTask將任務添加到隊列中,如果不能添加則拒絕;
- 如果當前運行的線程不是所綁定的線程則調用startThread方法為本EventExecutor綁定支撐線程。
@Override
public void execute(Runnable task) {
if (task == null) {
throw new NullPointerException("task");
}
boolean inEventLoop = inEventLoop();
addTask(task);
if (!inEventLoop) {
startThread();
if (isShutdown() && removeTask(task)) {
reject();
}
}
if (!addTaskWakesUp && wakesUpForTask(task)) {
wakeup(inEventLoop);
}
}
從下面的部分代碼可以看到,如果EventExecutor的狀態為ST_NOT_STARTED,那么先修改狀態然后調用doStartThread方法為本EventExecutor綁定線程,斷言指出此時thread必須為null,表明當前EventExecutor還未綁定任何線程。綁定任務交由線程池調度執行,線程池中執行該任務的線程被綁定到EventExecutor上,綁定的線程會運行SingleThreadEventExecutor的run抽象方法。
private void startThread() {
if (state == ST_NOT_STARTED) {
if (STATE_UPDATER.compareAndSet(this, ST_NOT_STARTED, ST_STARTED)) {
try {
doStartThread();
} catch (Throwable cause) {
STATE_UPDATER.set(this, ST_NOT_STARTED);
PlatformDependent.throwException(cause);
}
}
}
}
private void doStartThread() {
assert thread == null;
executor.execute(new Runnable() {
@Override
public void run() {
thread = Thread.currentThread();
if (interrupted) {
thread.interrupt();
}
boolean success = false;
updateLastExecutionTime();
try {
SingleThreadEventExecutor.this.run();
success = true;
} catch (Throwable t) {
logger.warn("Unexpected exception from an event executor: ", t);
} finally {
// 省略一些代碼
}
}
});
}
protected abstract void run();
從上述線程綁定過程不難理解《Netty實戰》3.1.2所述:
一個EventLoop在它的生命周期內只和一個線程綁定;
所有由EventLoop處理的I/O事件都將在它專有的線程上被處理。
SingleThreadEventLoop類
SingleThreadEventLoop類繼承了SingleThreadEventExecutor類并實現了EventLoop接口,增加了與通道注冊有關的register等方法,這正是前文末尾提到的將通道注冊到EventLoop上的關鍵。
@Override
public ChannelFuture register(Channel channel) {
return register(new DefaultChannelPromise(channel, this));
}
@Override
public ChannelFuture register(final ChannelPromise promise) {
ObjectUtil.checkNotNull(promise, "promise");
promise.channel().unsafe().register(this, promise);
return promise;
}
register方法會調用Channel接口的內部接口Unsafe的register方法,以服務端的NioServerSocketChannel為例,它的unsafe方法會返回AbstractNioMessageChannel類的內部類NioMessageUnsafe,其register方法定義在其父類AbstractUnsafe中:
public final void register(EventLoop eventLoop, final ChannelPromise promise) {
// 省略一些代碼
AbstractChannel.this.eventLoop = eventLoop;
if (eventLoop.inEventLoop()) {
register0(promise);
} else {
try {
eventLoop.execute(new Runnable() {
@Override
public void run() {
register0(promise);
}
});
} catch (Throwable t) {
// 省略一些代碼
}
}
}
AbstractChannel.this.eventLoop = eventLoop; 這一行將Channel注冊到EventLoop。此時不難理解《Netty實戰》3.1.2節所述:
一個Channel在它的生命周期內只注冊于一個EventLoop;
一個EventLoop可能會被分配給一個或多個Channel。
《Netty實戰》4.2節提到了Channel的線程安全性,這是因為對Channel的I/O操作都在Channel注冊的EventLoop上運行且一個EventLoop只和一個線程綁定:
Netty的Channel是線程安全的,因此你可以存儲一個到Channel的引用。
《Netty實戰》7.4.2節提到了一個潛在的陷阱:
需要注意的是,EventLoop的分配方式對ThreadLocal的使用的影響。因為一個EventLoop通常會被用于支撐多個Channel,所以對于所有相關的Channel來說,ThreadLocal都將是一樣的。
通道注冊
將通道注冊到EventLoop后接著會調用register0方法完成通道注冊過程,AbstractChannel的內部類AbstractUnsafe的register0方法如下:
private void register0(ChannelPromise promise) {
try {
// 省略一些代碼
boolean firstRegistration = neverRegistered;
doRegister();
neverRegistered = false;
registered = true;
pipeline.invokeHandlerAddedIfNeeded();
safeSetSuccess(promise);
pipeline.fireChannelRegistered();
// 省略一些代碼
} catch (Throwable t) {
// 省略一些代碼
}
}
- doRegister是AbstractChannel的方法,由子類覆蓋實現;
- NioServerSocketChannel的doRegister方法在父類AbstractNioChannel中定義如下,通道利用該方法注冊到NioEventLoop的Selector上并將通道自身關聯到結果鍵上。
@Override protected void doRegister() throws Exception { boolean selected = false; for (;;) { try { selectionKey = javaChannel().register(eventLoop().unwrappedSelector(), 0, this); return; } catch (CancelledKeyException e) { // 省略一些代碼 } } }
NioServerSocketChannel的感興趣集
由上面的方法可以看到,注冊時提供的感興趣集是0,但NioServerSocketChannel構造函數傳入的參數明明是SelectionKey.OP_ACCEPT,那么可連接是如何被設置到Selector的感興趣集上的呢?答案是在通道綁定本地地址時,這個問題留在后面的文章分析。
NioEventLoop類
NioEventLoop類繼承了SingleThreadEventLoop類,添加了與NIO和Selector相關的代碼,以實現將通道注冊到Selector上。
成員變量與構造函數
NioEventLoop類重要的成員變量和構造函數如下所示:
public final class NioEventLoop extends SingleThreadEventLoop {
private static final boolean DISABLE_KEYSET_OPTIMIZATION =
SystemPropertyUtil.getBoolean("io.netty.noKeySetOptimization", false);
// 省略一些代碼
/**
* The NIO {@link Selector}.
*/
private Selector selector;
private Selector unwrappedSelector;
private SelectedSelectionKeySet selectedKeys;
private final SelectorProvider provider;
/**
* Boolean that controls determines if a blocked Selector.select should
* break out of its selection process. In our case we use a timeout for
* the select method and the select method will block for that time unless
* waken up.
*/
private final AtomicBoolean wakenUp = new AtomicBoolean();
private final SelectStrategy selectStrategy;
private volatile int ioRatio = 50;
private int cancelledKeys;
private boolean needsToSelectAgain;
NioEventLoop(NioEventLoopGroup parent, Executor executor, SelectorProvider selectorProvider,
SelectStrategy strategy, RejectedExecutionHandler rejectedExecutionHandler) {
super(parent, executor, false, DEFAULT_MAX_PENDING_TASKS, rejectedExecutionHandler);
if (selectorProvider == null) {
throw new NullPointerException("selectorProvider");
}
if (strategy == null) {
throw new NullPointerException("selectStrategy");
}
provider = selectorProvider;
final SelectorTuple selectorTuple = openSelector();
selector = selectorTuple.selector;
unwrappedSelector = selectorTuple.unwrappedSelector;
selectStrategy = strategy;
}
}
- selector和unwrappedSelector分別表示優化過的Selector和未優化過的Selector,selectedKeys表示優化過的SelectionKey。Netty在該類中對Java NIO的Selector做了優化,可以通過設置系統屬性io.netty.noKeySetOptimization進行修改,設置為true、yes或者1關閉優化,設置為false、no或者0開啟優化,默認開啟優化。
- 如果啟用優化,那么selector和unwrappedSelector相同,selectedKeys為null;
- 如果關閉優化,那么selector和unwrappedSelector不同,selectedKeys不為null。
- 在構造函數的參數中,parent即為創建NioEventLoop的NioEventLoopGroup實例,其余均由NioEventLoopGroup的newChild方法提供,newChild提供的參數則是來自于NioEventLoopGroup的構造函數。
多路復用
上文提到通道利用AbstractNioChannel類的doRegister方法注冊到NioEventLoop的Selector上并將通道自身關聯到結果鍵上,那么NioEventLoop的Selector是如何處理通道的呢?在分析SingleThreadEventExecutor類時,我們知道綁定的線程會運行SingleThreadEventExecutor的run抽象方法,NioEventLoop恰恰重寫了SingleThreadEventExecutor類的抽象方法run:
@Override
protected void run() {
for (;;) {
try {
switch (selectStrategy.calculateStrategy(selectNowSupplier, hasTasks())) {
case SelectStrategy.CONTINUE:
continue;
case SelectStrategy.SELECT:
select(wakenUp.getAndSet(false));
if (wakenUp.get()) {
selector.wakeup();
}
// fall through
default:
}
cancelledKeys = 0;
needsToSelectAgain = false;
final int ioRatio = this.ioRatio;
if (ioRatio == 100) {
try {
processSelectedKeys();
} finally {
// Ensure we always run tasks.
runAllTasks();
}
} else {
final long ioStartTime = System.nanoTime();
try {
processSelectedKeys();
} finally {
// Ensure we always run tasks.
final long ioTime = System.nanoTime() - ioStartTime;
runAllTasks(ioTime * (100 - ioRatio) / ioRatio);
}
}
} catch (Throwable t) {
handleLoopException(t);
}
// 省略一些代碼
}
}
- 死循環會使支撐線程(I/O線程)一直運行run方法;
- select(wakenUp.getAndSet(false)); 執行真正的Java NIO select操作;
- processSelectedKeys方法處理select返回的已選擇鍵。
processSelectedKeys方法代碼如下所示。該方法判斷是否啟用了Selector優化,如果啟用則交由processSelectedKeysOptimized方法,否則交由processSelectedKeysPlain方法。
private void processSelectedKeys() {
if (selectedKeys != null) {
processSelectedKeysOptimized();
} else {
processSelectedKeysPlain(selector.selectedKeys());
}
}
以processSelectedKeysPlain為例,上文提到通道會將自己關聯到結果鍵上,用處是在這里用SelectionKey的attachment方法取出通道:
private void processSelectedKeysPlain(Set<SelectionKey> selectedKeys) {
if (selectedKeys.isEmpty()) {
return;
}
Iterator<SelectionKey> i = selectedKeys.iterator();
for (;;) {
final SelectionKey k = i.next();
final Object a = k.attachment();
i.remove();
if (a instanceof AbstractNioChannel) {
processSelectedKey(k, (AbstractNioChannel) a);
} else {
@SuppressWarnings("unchecked")
NioTask<SelectableChannel> task = (NioTask<SelectableChannel>) a;
processSelectedKey(k, task);
}
if (!i.hasNext()) {
break;
}
if (needsToSelectAgain) {
selectAgain();
selectedKeys = selector.selectedKeys();
// Create the iterator again to avoid ConcurrentModificationException
if (selectedKeys.isEmpty()) {
break;
} else {
i = selectedKeys.iterator();
}
}
}
}
通道對象被取出后便可以根據按位與執行相應的操作,當通道可讀或者可連接時,調用Unsafe的read方法,其他操作類似:
private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();
// 省略一些代碼
try {
int readyOps = k.readyOps();
// We first need to call finishConnect() before try to trigger a read(...) or write(...) as otherwise
// the NIO JDK channel implementation may throw a NotYetConnectedException.
if ((readyOps & SelectionKey.OP_CONNECT) != 0) {
// remove OP_CONNECT as otherwise Selector.select(..) will always return without blocking
// See https://github.com/netty/netty/issues/924
int ops = k.interestOps();
ops &= ~SelectionKey.OP_CONNECT;
k.interestOps(ops);
unsafe.finishConnect();
}
// Process OP_WRITE first as we may be able to write some queued buffers and so free memory.
if ((readyOps & SelectionKey.OP_WRITE) != 0) {
// Call forceFlush which will also take care of clear the OP_WRITE once there is nothing left to write
ch.unsafe().forceFlush();
}
// Also check for readOps of 0 to workaround possible JDK bug which may otherwise lead
// to a spin loop
if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
unsafe.read();
}
} catch (CancelledKeyException ignored) {
unsafe.close(unsafe.voidPromise());
}
}
實例 NioServerSocketChannel
以NioServerSocketChannel為例,當可連接時會調用AbstractNioMessageChannel的內部類NioMessageUnsafe的read方法:
@Override
public void read() {
// 省略一些代碼
try {
try {
do {
int localRead = doReadMessages(readBuf);
if (localRead == 0) {
break;
}
if (localRead < 0) {
closed = true;
break;
}
allocHandle.incMessagesRead(localRead);
} while (allocHandle.continueReading());
} catch (Throwable t) {
exception = t;
}
int size = readBuf.size();
for (int i = 0; i < size; i ++) {
readPending = false;
pipeline.fireChannelRead(readBuf.get(i));
}
readBuf.clear();
allocHandle.readComplete();
pipeline.fireChannelReadComplete();
// 省略一些代碼
} finally {
// 省略一些代碼
}
}
doReadMessages是AbstractNioMessageChannel類的抽象方法,NioServerSocketChannel重寫了該方法:
@Override
protected int doReadMessages(List<Object> buf) throws Exception {
SocketChannel ch = SocketUtils.accept(javaChannel());
try {
if (ch != null) {
buf.add(new NioSocketChannel(this, ch));
return 1;
}
} catch (Throwable t) {
// 省略一些代碼
}
return 0;
}
doReadMessages方法首先接受了連接,然后將已連接套接字通道封裝成消息加到了buf中,返回了數量1。在上文的read方法中,pipeline.fireChannelRead(readBuf.get(i)) 觸發了入站可讀事件,這也就可以解釋之前提到的ServerBootstrapAcceptor的channelRead方法的消息參數為什么是一個通道的問題了。