Netty學習 - EventLoop

本文以常見的NioEventLoop為切入點分析Netty的EventLoop,NioEventLoop的類層次結構如下圖所示,下面將按照類層次結構自底向上依次分析。


NioEventLoop.png

AbstractEventExecutor類

AbstractEventExecutor類實現了EventExecutor接口,比較重要的是parent成員變量,用于引用該EventExecutor所屬的EventExecutorGroup,以schedule開頭的調度方法都拋出了UnsupportedOperationException,子類需要覆蓋這些方法。

public abstract class AbstractEventExecutor extends AbstractExecutorService implements EventExecutor {
    private final EventExecutorGroup parent;
    // 省略一些代碼

    protected AbstractEventExecutor() {
        this(null);
    }

    protected AbstractEventExecutor(EventExecutorGroup parent) {
        this.parent = parent;
    }

    @Override
    public EventExecutorGroup parent() {
        return parent;
    }

    @Override
    public EventExecutor next() {
        return this;
    }

    @Override
    public boolean inEventLoop() {
        return inEventLoop(Thread.currentThread());
    }

   @Override
    public ScheduledFuture<?> schedule(Runnable command, long delay,
                                       TimeUnit unit) {
        throw new UnsupportedOperationException();
    }
    // 省略一些代碼
}

AbstractScheduledEventExecutor類

AbstractScheduledEventExecutor類繼承了AbstractEventExecutor類,實現了調度方法以支持任務調度。

SingleThreadEventExecutor類

SingleThreadEventExecutor類繼承了AbstractScheduledEventExecutor類,從類名可以看出該EventExecutor是單線程的。

成員變量和構造函數

該類重要的成員變量和構造函數如下所示:

public abstract class SingleThreadEventExecutor extends AbstractScheduledEventExecutor implements OrderedEventExecutor {
    static final int DEFAULT_MAX_PENDING_EXECUTOR_TASKS = Math.max(16,
            SystemPropertyUtil.getInt("io.netty.eventexecutor.maxPendingTasks", Integer.MAX_VALUE));
    private final Queue<Runnable> taskQueue;
    private volatile Thread thread;
    private volatile ThreadProperties threadProperties;
    private final Executor executor;
    private volatile boolean interrupted;
    // 省略一些代碼
    
    protected SingleThreadEventExecutor(
            EventExecutorGroup parent, ThreadFactory threadFactory, boolean addTaskWakesUp) {
        this(parent, new ThreadPerTaskExecutor(threadFactory), addTaskWakesUp);
    }

    protected SingleThreadEventExecutor(
            EventExecutorGroup parent, ThreadFactory threadFactory,
            boolean addTaskWakesUp, int maxPendingTasks, RejectedExecutionHandler rejectedHandler) {
        this(parent, new ThreadPerTaskExecutor(threadFactory), addTaskWakesUp, maxPendingTasks, rejectedHandler);
    }

    protected SingleThreadEventExecutor(EventExecutorGroup parent, Executor executor, boolean addTaskWakesUp) {
        this(parent, executor, addTaskWakesUp, DEFAULT_MAX_PENDING_EXECUTOR_TASKS, RejectedExecutionHandlers.reject());
    }

    protected SingleThreadEventExecutor(EventExecutorGroup parent, Executor executor,
                                        boolean addTaskWakesUp, int maxPendingTasks,
                                        RejectedExecutionHandler rejectedHandler) {
        super(parent);
        this.addTaskWakesUp = addTaskWakesUp;
        this.maxPendingTasks = Math.max(16, maxPendingTasks);
        this.executor = ObjectUtil.checkNotNull(executor, "executor");
        taskQueue = newTaskQueue(this.maxPendingTasks);
        rejectedExecutionHandler = ObjectUtil.checkNotNull(rejectedHandler, "rejectedHandler");
    }
    // 省略一些代碼
}
  • taskQueue是任務隊列,用來存放需要調度執行的任務;
  • thread用來引用支撐該EventExecutor的線程,用來處理I/O事件和執行任務,叫支撐線程或者I/O線程均可;
  • executor用來引用線程池,thread所引用的線程即來自這里。

inEventLoop方法

其父類AbstractEventExecutor類中的inEventLoop()實現調用了inEventLoop(Thread.currentThread()),以下是其實現:

@Override
public boolean inEventLoop(Thread thread) {
    return thread == this.thread;
}

可以看到是用該EventExecutor所綁定的線程與參數去做比較,所以inEventLoop()的語義較為明晰:如果當前運行的線程是EventExecutor的支撐線程則返回true,否則返回false。這一方法在DefaultChannelPipeline和AbstractChannelHandlerContext類中意義重大,如《Netty實戰》7.4.1節所述:

Netty線程模型的卓越性能取決于對于當前執行線程的身份的確定,也就是說,確定它是否是分配給當前Channel以及它的EventLoop的那一個線程。
如果(當前)調用線程正是支撐EventLoop的線程,那么所提交的代碼塊將會被(直接)執行。否則,EventLoop將調度該任務以便稍后執行,并將它放入到內部隊列中。當EventLoop下次處理它的事件時,它會執行隊列中的那些任務。這也就解釋了任何的線程是如何與Channel直接交互而無需在ChannelHandler中進行額外同步的。

任務執行與線程綁定

execute方法實現了Java并發包Executor的接口方法,用來執行任務(如通道注冊等):

  • addTask將任務添加到隊列中,如果不能添加則拒絕;
  • 如果當前運行的線程不是所綁定的線程則調用startThread方法為本EventExecutor綁定支撐線程。
@Override
public void execute(Runnable task) {
    if (task == null) {
        throw new NullPointerException("task");
    }

    boolean inEventLoop = inEventLoop();
    addTask(task);
    if (!inEventLoop) {
        startThread();
        if (isShutdown() && removeTask(task)) {
            reject();
        }
    }

    if (!addTaskWakesUp && wakesUpForTask(task)) {
        wakeup(inEventLoop);
    }
}

從下面的部分代碼可以看到,如果EventExecutor的狀態為ST_NOT_STARTED,那么先修改狀態然后調用doStartThread方法為本EventExecutor綁定線程,斷言指出此時thread必須為null,表明當前EventExecutor還未綁定任何線程。綁定任務交由線程池調度執行,線程池中執行該任務的線程被綁定到EventExecutor上,綁定的線程會運行SingleThreadEventExecutor的run抽象方法。

private void startThread() {
    if (state == ST_NOT_STARTED) {
        if (STATE_UPDATER.compareAndSet(this, ST_NOT_STARTED, ST_STARTED)) {
            try {
                doStartThread();
            } catch (Throwable cause) {
                STATE_UPDATER.set(this, ST_NOT_STARTED);
                PlatformDependent.throwException(cause);
            }
        }
    }
}

private void doStartThread() {
    assert thread == null;
    executor.execute(new Runnable() {
        @Override
        public void run() {
            thread = Thread.currentThread();
            if (interrupted) {
                thread.interrupt();
            }

            boolean success = false;
            updateLastExecutionTime();
            try {
                SingleThreadEventExecutor.this.run();
                success = true;
            } catch (Throwable t) {
                logger.warn("Unexpected exception from an event executor: ", t);
            } finally {
                // 省略一些代碼
            }
        }
    });
}

protected abstract void run();

從上述線程綁定過程不難理解《Netty實戰》3.1.2所述:

一個EventLoop在它的生命周期內只和一個線程綁定;
所有由EventLoop處理的I/O事件都將在它專有的線程上被處理。

SingleThreadEventLoop類

SingleThreadEventLoop類繼承了SingleThreadEventExecutor類并實現了EventLoop接口,增加了與通道注冊有關的register等方法,這正是前文末尾提到的將通道注冊到EventLoop上的關鍵。

@Override
public ChannelFuture register(Channel channel) {
    return register(new DefaultChannelPromise(channel, this));
}

@Override
public ChannelFuture register(final ChannelPromise promise) {
    ObjectUtil.checkNotNull(promise, "promise");
    promise.channel().unsafe().register(this, promise);
    return promise;
}

register方法會調用Channel接口的內部接口Unsafe的register方法,以服務端的NioServerSocketChannel為例,它的unsafe方法會返回AbstractNioMessageChannel類的內部類NioMessageUnsafe,其register方法定義在其父類AbstractUnsafe中:

public final void register(EventLoop eventLoop, final ChannelPromise promise) {
    // 省略一些代碼
    AbstractChannel.this.eventLoop = eventLoop;
    if (eventLoop.inEventLoop()) {
        register0(promise);
    } else {
        try {
            eventLoop.execute(new Runnable() {
                @Override
                public void run() {
                    register0(promise);
                }
            });
        } catch (Throwable t) {
            // 省略一些代碼
        }
    }
}

AbstractChannel.this.eventLoop = eventLoop; 這一行將Channel注冊到EventLoop。此時不難理解《Netty實戰》3.1.2節所述:

一個Channel在它的生命周期內只注冊于一個EventLoop;
一個EventLoop可能會被分配給一個或多個Channel。

《Netty實戰》4.2節提到了Channel的線程安全性,這是因為對Channel的I/O操作都在Channel注冊的EventLoop上運行且一個EventLoop只和一個線程綁定:

Netty的Channel是線程安全的,因此你可以存儲一個到Channel的引用。

《Netty實戰》7.4.2節提到了一個潛在的陷阱:

需要注意的是,EventLoop的分配方式對ThreadLocal的使用的影響。因為一個EventLoop通常會被用于支撐多個Channel,所以對于所有相關的Channel來說,ThreadLocal都將是一樣的。

通道注冊

將通道注冊到EventLoop后接著會調用register0方法完成通道注冊過程,AbstractChannel的內部類AbstractUnsafe的register0方法如下:

private void register0(ChannelPromise promise) {
    try {
        // 省略一些代碼
        boolean firstRegistration = neverRegistered;
        doRegister();
        neverRegistered = false;
        registered = true;
        pipeline.invokeHandlerAddedIfNeeded();
        safeSetSuccess(promise);
        pipeline.fireChannelRegistered();
        // 省略一些代碼
    } catch (Throwable t) {
        // 省略一些代碼
    }
}
  • doRegister是AbstractChannel的方法,由子類覆蓋實現;
  • NioServerSocketChannel的doRegister方法在父類AbstractNioChannel中定義如下,通道利用該方法注冊到NioEventLoop的Selector上并將通道自身關聯到結果鍵上。
    @Override
    protected void doRegister() throws Exception {
        boolean selected = false;
        for (;;) {
            try {
                selectionKey = javaChannel().register(eventLoop().unwrappedSelector(), 0, this);
                return;
            } catch (CancelledKeyException e) {
                // 省略一些代碼
            }
        }
    }
    

NioServerSocketChannel的感興趣集

由上面的方法可以看到,注冊時提供的感興趣集是0,但NioServerSocketChannel構造函數傳入的參數明明是SelectionKey.OP_ACCEPT,那么可連接是如何被設置到Selector的感興趣集上的呢?答案是在通道綁定本地地址時,這個問題留在后面的文章分析。

NioEventLoop類

NioEventLoop類繼承了SingleThreadEventLoop類,添加了與NIO和Selector相關的代碼,以實現將通道注冊到Selector上。

成員變量與構造函數

NioEventLoop類重要的成員變量和構造函數如下所示:

public final class NioEventLoop extends SingleThreadEventLoop {
    private static final boolean DISABLE_KEYSET_OPTIMIZATION =
            SystemPropertyUtil.getBoolean("io.netty.noKeySetOptimization", false);
    // 省略一些代碼
    /**
     * The NIO {@link Selector}.
     */
    private Selector selector;
    private Selector unwrappedSelector;
    private SelectedSelectionKeySet selectedKeys;

    private final SelectorProvider provider;

    /**
     * Boolean that controls determines if a blocked Selector.select should
     * break out of its selection process. In our case we use a timeout for
     * the select method and the select method will block for that time unless
     * waken up.
     */
    private final AtomicBoolean wakenUp = new AtomicBoolean();

    private final SelectStrategy selectStrategy;

    private volatile int ioRatio = 50;
    private int cancelledKeys;
    private boolean needsToSelectAgain;

    NioEventLoop(NioEventLoopGroup parent, Executor executor, SelectorProvider selectorProvider,
                 SelectStrategy strategy, RejectedExecutionHandler rejectedExecutionHandler) {
        super(parent, executor, false, DEFAULT_MAX_PENDING_TASKS, rejectedExecutionHandler);
        if (selectorProvider == null) {
            throw new NullPointerException("selectorProvider");
        }
        if (strategy == null) {
            throw new NullPointerException("selectStrategy");
        }
        provider = selectorProvider;
        final SelectorTuple selectorTuple = openSelector();
        selector = selectorTuple.selector;
        unwrappedSelector = selectorTuple.unwrappedSelector;
        selectStrategy = strategy;
    }
}
  1. selector和unwrappedSelector分別表示優化過的Selector和未優化過的Selector,selectedKeys表示優化過的SelectionKey。Netty在該類中對Java NIO的Selector做了優化,可以通過設置系統屬性io.netty.noKeySetOptimization進行修改,設置為true、yes或者1關閉優化,設置為false、no或者0開啟優化,默認開啟優化。
    • 如果啟用優化,那么selector和unwrappedSelector相同,selectedKeys為null;
    • 如果關閉優化,那么selector和unwrappedSelector不同,selectedKeys不為null。
  2. 在構造函數的參數中,parent即為創建NioEventLoop的NioEventLoopGroup實例,其余均由NioEventLoopGroup的newChild方法提供,newChild提供的參數則是來自于NioEventLoopGroup的構造函數。

多路復用

上文提到通道利用AbstractNioChannel類的doRegister方法注冊到NioEventLoop的Selector上并將通道自身關聯到結果鍵上,那么NioEventLoop的Selector是如何處理通道的呢?在分析SingleThreadEventExecutor類時,我們知道綁定的線程會運行SingleThreadEventExecutor的run抽象方法,NioEventLoop恰恰重寫了SingleThreadEventExecutor類的抽象方法run:

@Override
protected void run() {
    for (;;) {
        try {
            switch (selectStrategy.calculateStrategy(selectNowSupplier, hasTasks())) {
                case SelectStrategy.CONTINUE:
                    continue;
                case SelectStrategy.SELECT:
                    select(wakenUp.getAndSet(false));
                    if (wakenUp.get()) {
                        selector.wakeup();
                    }
                    // fall through
                default:
            }

            cancelledKeys = 0;
            needsToSelectAgain = false;
            final int ioRatio = this.ioRatio;
            if (ioRatio == 100) {
                try {
                    processSelectedKeys();
                } finally {
                    // Ensure we always run tasks.
                    runAllTasks();
                }
            } else {
                final long ioStartTime = System.nanoTime();
                try {
                    processSelectedKeys();
                } finally {
                    // Ensure we always run tasks.
                    final long ioTime = System.nanoTime() - ioStartTime;
                    runAllTasks(ioTime * (100 - ioRatio) / ioRatio);
                }
            }
        } catch (Throwable t) {
            handleLoopException(t);
        }
        // 省略一些代碼
    }
}
  • 死循環會使支撐線程(I/O線程)一直運行run方法;
  • select(wakenUp.getAndSet(false)); 執行真正的Java NIO select操作;
  • processSelectedKeys方法處理select返回的已選擇鍵。

processSelectedKeys方法代碼如下所示。該方法判斷是否啟用了Selector優化,如果啟用則交由processSelectedKeysOptimized方法,否則交由processSelectedKeysPlain方法。

private void processSelectedKeys() {
    if (selectedKeys != null) {
        processSelectedKeysOptimized();
    } else {
        processSelectedKeysPlain(selector.selectedKeys());
    }
}

以processSelectedKeysPlain為例,上文提到通道會將自己關聯到結果鍵上,用處是在這里用SelectionKey的attachment方法取出通道:

private void processSelectedKeysPlain(Set<SelectionKey> selectedKeys) {
    if (selectedKeys.isEmpty()) {
        return;
    }

    Iterator<SelectionKey> i = selectedKeys.iterator();
    for (;;) {
        final SelectionKey k = i.next();
        final Object a = k.attachment();
        i.remove();

        if (a instanceof AbstractNioChannel) {
            processSelectedKey(k, (AbstractNioChannel) a);
        } else {
            @SuppressWarnings("unchecked")
            NioTask<SelectableChannel> task = (NioTask<SelectableChannel>) a;
            processSelectedKey(k, task);
        }

        if (!i.hasNext()) {
            break;
        }

        if (needsToSelectAgain) {
            selectAgain();
            selectedKeys = selector.selectedKeys();

            // Create the iterator again to avoid ConcurrentModificationException
            if (selectedKeys.isEmpty()) {
                break;
            } else {
                i = selectedKeys.iterator();
            }
        }
    }
}

通道對象被取出后便可以根據按位與執行相應的操作,當通道可讀或者可連接時,調用Unsafe的read方法,其他操作類似:

private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
    final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();
    // 省略一些代碼

    try {
        int readyOps = k.readyOps();
        // We first need to call finishConnect() before try to trigger a read(...) or write(...) as otherwise
        // the NIO JDK channel implementation may throw a NotYetConnectedException.
        if ((readyOps & SelectionKey.OP_CONNECT) != 0) {
            // remove OP_CONNECT as otherwise Selector.select(..) will always return without blocking
            // See https://github.com/netty/netty/issues/924
            int ops = k.interestOps();
            ops &= ~SelectionKey.OP_CONNECT;
            k.interestOps(ops);

            unsafe.finishConnect();
        }

        // Process OP_WRITE first as we may be able to write some queued buffers and so free memory.
        if ((readyOps & SelectionKey.OP_WRITE) != 0) {
            // Call forceFlush which will also take care of clear the OP_WRITE once there is nothing left to write
            ch.unsafe().forceFlush();
        }

        // Also check for readOps of 0 to workaround possible JDK bug which may otherwise lead
        // to a spin loop
        if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
            unsafe.read();
        }
    } catch (CancelledKeyException ignored) {
        unsafe.close(unsafe.voidPromise());
    }
}

實例 NioServerSocketChannel

以NioServerSocketChannel為例,當可連接時會調用AbstractNioMessageChannel的內部類NioMessageUnsafe的read方法:

@Override
public void read() {
    // 省略一些代碼
    try {
        try {
            do {
                int localRead = doReadMessages(readBuf);
                if (localRead == 0) {
                    break;
                }
                if (localRead < 0) {
                    closed = true;
                    break;
                }
                allocHandle.incMessagesRead(localRead);
            } while (allocHandle.continueReading());
        } catch (Throwable t) {
            exception = t;
        }
        int size = readBuf.size();
        for (int i = 0; i < size; i ++) {
            readPending = false;
            pipeline.fireChannelRead(readBuf.get(i));
        }
        readBuf.clear();
        allocHandle.readComplete();
        pipeline.fireChannelReadComplete();
        // 省略一些代碼
    } finally {
        // 省略一些代碼
    }
}

doReadMessages是AbstractNioMessageChannel類的抽象方法,NioServerSocketChannel重寫了該方法:

@Override
protected int doReadMessages(List<Object> buf) throws Exception {
    SocketChannel ch = SocketUtils.accept(javaChannel());
    try {
        if (ch != null) {
            buf.add(new NioSocketChannel(this, ch));
            return 1;
        }
    } catch (Throwable t) {
        // 省略一些代碼
    }
    return 0;
}

doReadMessages方法首先接受了連接,然后將已連接套接字通道封裝成消息加到了buf中,返回了數量1。在上文的read方法中,pipeline.fireChannelRead(readBuf.get(i)) 觸發了入站可讀事件,這也就可以解釋之前提到的ServerBootstrapAcceptor的channelRead方法的消息參數為什么是一個通道的問題了。

最后編輯于
?著作權歸作者所有,轉載或內容合作請聯系作者
平臺聲明:文章內容(如有圖片或視頻亦包括在內)由作者上傳并發布,文章內容僅代表作者本人觀點,簡書系信息發布平臺,僅提供信息存儲服務。

推薦閱讀更多精彩內容