HBase 1.2.0源碼系列:HBase RPC 通信(上)

HBase 中 HMaster、HRegionServer 和 Client 之間的通信使用了兩個技術,Google Protobuf RPC 和 Java NIO。

主要代碼位置:

.
|—— hbase-client
|    |—— org.apache.hadoop.hbase.ipc
|    |—— org.apache.hadoop.hbase.client
|—— hbase-server
|    |—— org.apache.hadoop.hbase.ipc
|    |—— org.apache.hadoop.hbase.master
|    |—— org.apache.hadoop.hbase.regionserver
|—— hbase-protocal 
|    |—— org.apache.hadoop.hbase.protobuf.generated
...

HBase RPC

什么是 PRC

RPC(Remote Procedure Call)即遠程過程調用。對于本地調用,定義好一個函數以后,程序的其他部分通過調用該函數,就可以返回想要的結果。RPC 的區別就是函數定義和函數調用位于不同的機器(大部分情況),因為涉及到不同的機器,所以 RPC 相比較本地函數調用多了通信部分。主要涉及到兩個角色:調用方(client)和函數定義實現(server),RPC 調用的流程如下面圖所示:

RPC

HBase Server 端主要類關系:

[圖片上傳失敗...(image-6f67b9-1560062040880)]

HMaster 繼承 HRegionServer,rpcService 提供 RPC Server 端實現(HRegionServier 中由 RSRpcService 實現,HMaster 中由 MasterRpcService 實現),rpcServer 是具體的 RPC Server 實現(實現 RpcServerInterface 接口),Listener 線程負責監聽請求,Resonder 線程負責發送請求結果。

HBase Client 端主要類關系:

HBase Client

HBase Client 訪問 HBase 需要先創建 HConnection,Connection 中的 rpcClient(RpcClient 接口,RpcClientImpl 是實現類)表示 Rpc Client 端實現,由 RpcClientFactory 創建。

RPC 初始化

在 HRegionServer 啟動類的源碼中,有以下代碼,分別初始化 RpcServer 和 RpcClient:

public HRegionServer(Configuration conf, CoordinatedStateManager csm)
      throws IOException, InterruptedException {
    // ...
    
    rpcServices = createRpcServices();
    
    // ...
}

private void initializeThreads() {
    // ...
    
    rpcClient = RpcClientFactory.createClient(conf, clusterId, new InetSocketAddress(
        rpcServices.isa.getAddress(), 0), clusterConnection.getConnectionMetrics());    
    
    // ...
}

RPC Server

先來看以下 RPC Server 端的一些實現,從一些重要類的初始化開始

初始化

createRpcServices()

createRpcServices() 方法是初始化 RPC Server 端實現的入口:

class HRegionServer {
    protected RSRpcServices createRpcServices() throws IOException {
        return new RSRpcServices(this);
    }
}

class HMaseter implements HRegionServer {
    @Override
    protected RSRpcServices createRpcServices() throws IOException {
        return new MasterRpcServices(this);
    }
}

RSRpcServices#Constructor

MasterRpcServices 的構造方法調用父類 RSRpcServices 的構造方法:

public MasterRpcServices(HMaster m) throws IOException {
    super(m);
    // set HMaster
    master = m;
}
  
public RSRpcServices(HRegionServer rs) throws IOException {
    // set HRegionServer
    regionServer = rs;

    // 初始化 RpcSchedulerFactory
    // 反射 hbase.region.server.rpc.scheduler.factory.class 指定的類
    // 默認使用 SimpleRpcSchedulerFactory
    RpcSchedulerFactory rpcSchedulerFactory = ...

    // ...
    
    // 優先級函數(調度請求時使用)
    priority = createPriority();
    
    // 設置訪問時的重試次數
    ConnectionUtils.setServerSideHConnectionRetriesConfig(rs.conf, name, LOG);
    
    // 初始化 RpcServer
    try {
        rpcServer = new RpcServer(rs, name, getServices(),
            bindAddress,
            rs.conf,
            rpcSchedulerFactory.create(rs.conf, this, rs));
    } catch (BindException be) {
        // throw Execption
    }

    // 初始化配置信息
    // hbase.client.scanner.timeout.period  # Client 端 Scan Timeout 
    // hbase.server.scanner.max.result.size Scan    # Scan 獲取的最大條目數 
    // hbase.rpc.timeout    # RPC 處理請求 Timeout 時間
    // hbase.region.server.rpc.minimum.scan.time.limit.delta    #
    scannerLeaseTimeoutPeriod = rs.conf.getInt(...);
    maxScannerResultSize = rs.conf.getLong(...);
    rpcTimeout = rs.conf.getInt(...);
    minimumScanTimeLimitDelta = rs.conf.getLong(...);

}

RpcServer#Constructor

RpcServer 實現了 RpcServerInterface 接口,構造函數:

public RpcServer(final Server server, final String name,
      final List<BlockingServiceAndInterface> services,
      final InetSocketAddress bindAddress, Configuration conf,
      RpcScheduler scheduler)
      throws IOException {
    
    // 初始化屬性 ...

    // 設置 Listener
    listener = new Listener(name);

    // 設置 Responder
    responder = new Responder();

    // 設置 Scheduler(調用方通過 RpcSchedulerFactory 創建)
    this.scheduler = scheduler;
    this.scheduler.init(new RpcSchedulerContext(this));
}

主要處理角色

Rpc Server 監控、讀取、請求基于 Reactor 模式,主要流程如下圖:


image

Listener

Listener 負責監聽請求,對于獲取到的請求,交由 Reader 負責讀取:

private class Listener extends Thread {

    private ServerSocketChannel acceptChannel = null; 
    private Selector selector = null; 
    private Reader[] readers = null;
    private ExecutorService readPool;

    public Listener(final String name) throws IOException {
        // ...

        // 創建非阻塞的 ServerSocketChannel
        acceptChannel = ServerSocketChannel.open();
        acceptChannel.configureBlocking(false);
        
        // 綁定 Socket 到 RpcServer#bingAddress
        bind(acceptChannel.socket(), bindAddress, backlogLength);
        
        // 創建 selector
        selector = Selector.open();

        // 初始化 Reader ThreadPool
        readers = new Reader[readThreads];
        readPool = Executors.newFixedThreadPool(readThreads,
            new ThreadFactoryBuilder()
                .setNameFormat(...)
                .setDaemon(true)
                .build());
        
        for (int i = 0; i < readThreads; ++i) {
            Reader reader = new Reader();
            readers[i] = reader;
            readPool.execute(reader);
        }

        // 注冊 selector
        acceptChannel.register(selector, SelectionKey.OP_ACCEPT);
    }
}

Reader

處理請求的邏輯在 Reader 中,生成 Call 對象交由 RpcSchedule 進行分發

private class Reader implements Runnable {
    private final Selector readSelector;

    @Override
    public void run() {
        try {
            doRunLoop();
        } finally {
            // close readSelector
        }
    }

    private synchronized void doRunLoop() {
        while (running) {
            try {
                // 線程阻塞,知道有請求到來
                readSelector.select();
                
                Iterator<SelectionKey> iter = readSelector.selectedKeys().iterator();
                while (iter.hasNext()) {
                    SelectionKey key = iter.next();
                    iter.remove();
                    if (key.isValid()) {
                        if (key.isReadable()) {
                            // 請求有效,進行處理
                            doRead(key);
                        }
                    }
                }
            } catch (Exception e) {
                // ...
            }
        }
    }
}

doRead() 方法在 Listener 中,由 Connection 對象處理,生成 Call,并包裝為 CallRunner 交給 Scheduler

class Listener {
    void doRead(SelectionKey key) throws InterruptedException {
        Connection c = (Connection) key.attachment();
    
        try {
            count = c.readAndProcess();
        } catch (Exception e) {
            // ...
        }
    }
}

class Connection {
    protected void processRequest(byte[] buf) throws IOException, InterruptedException {
        // ...
        
        // Dispatches an RPC request asynchronously
        if (!scheduler.dispatch(new CallRunner(RpcServer.this, call))) {
            
            // ...
            responder.doRespond(call);
        }
    }
}

Scheduler

Scheduler 是一個生產者消費者模型,內部有一個隊列緩存請求,另外有一些線程負責從隊列中拉取消息進行分發

Scheduler 默認實現為 SimpleRpcScheduler(HBase 提供的另一種實現為 FifoRpcScheduler),包含三個 RpcExecutor(callExecutor、priorityExecutor、replicationExecutor)

public class SimpleRpcScheduler extends RpcScheduler {

    /**
     * callExecutor = RWQueueRpcExecutor or BalancedQueueRpcExecutor
     * priorityExecutor = BalancedQueueRpcExecutor or NULL
     * replicationExecutor = BalancedQueueRpcExecutor or NULL
     **/
    private final RpcExecutor callExecutor;
    private final RpcExecutor priorityExecutor;
    private final RpcExecutor replicationExecutor;
    
    // 分發請求
    @Override
    public boolean dispatch(CallRunner callTask) throws InterruptedException {
        
        // 選擇不同的 Executor 處理
        // 大部分基于的請求都是通過 callExecutor 來執行
        if (priorityExecutor != null && level > highPriorityLevel) {
            return priorityExecutor.dispatch(callTask);
        } else if (replicationExecutor != null && level == HConstants.REPLICATION_QOS) {
            return replicationExecutor.dispatch(callTask);
        } else {
            return callExecutor.dispatch(callTask);
        }
    }
}

RpcExecutor

阻塞隊列

RpcExecutor 的實現類 RWQueueRpcExecutor 使用阻塞隊列緩存消息(BalancedQueueRpcExecutor 實現類似):

class RWQueueRpcExecutor extends RpcExecutor {
    private final List<BlockingQueue<CallRunner>> queues;
    
    @Override
    public boolean dispatch(final CallRunner callTask) throws InterruptedException {
        // 進入隊列
        return queues.get(queueIndex).offer(callTask);
    }
}
消費線程

RpcExecutor 處理的具體邏輯在 consumerLoop() 方法中,從阻塞隊列中取出 CallRunner 對象,并執行:

public abstract class RpcExecutor {

    // 啟動多線程處理
    protected void startHandlers(final String nameSuffix, final int numHandlers,
        final List<BlockingQueue<CallRunner>> callQueues,
        final int qindex, final int qsize, final int port) {
        
        for (int i = 0; i < numHandlers; i++) {
            final int index = qindex + (i % qsize);
            Thread t = new Thread(new Runnable() {
                @Override
                public void run() {
                    consumerLoop(callQueues.get(index));
                }
            });
            t.start();
        }
    }

    // 核心處理邏輯(省略部分代碼)
    protected void consumerLoop(final BlockingQueue<CallRunner> myQueue) {
        try {
            while (running) {
                try {
                    // 如果 BlockingQueue 中沒有數據,會在此阻塞
                    CallRunner task = myQueue.take();
                    try {
                        // 執行請求
                        task.run();
                    } catch (Throwable e) {
                        // throw or abort
                    } 
                } catch (InterruptedException e) {
                    interrupted = true;
                }
            }
        } finally {
            if (interrupted) {
                Thread.currentThread().interrupt();
            }
        }
    }
}
CallRunner & Call
image

CallRunner 和 Call 關鍵代碼如下:

class CallRunner {
    private Call call;

    public void run() {
        // ...        
        
        Pair<Message, CellScanner> resultPair = null;
        try {
            // make the call
            resultPair = this.rpcServer.call(call.service, call.md, call.param, call.cellScanner,
                call.timestamp, this.status);
        } catch (Throwable e) {
            // ...
        } 
        
        // Call to respond
        call.sendResponseIfReady();
    }
}

class Call {
    public synchronized void sendResponseIfReady() throws IOException {
        // 結果返回給 Client
        this.responder.doRespond(this);
    }
}

調用執行方法在 RpcServer#call() 方法中:

class RpcServer {

    @Override
    public Pair<Message, CellScanner> call(BlockingService service, MethodDescriptor md,
        Message param, CellScanner cellScanner, long receiveTime, MonitoredRPCHandler status)
        throws IOException {
    
    try {
        
        // call method
        // com.google.protobuf#BlockingService
        Message result = service.callBlockingMethod(md, controller, param);
      
        if (tooSlow || tooLarge) {
            // logging
        }
        return new Pair<Message, CellScanner>(result, controller.cellScanner());
    } catch (Throwable e) {
        // ...
    }
  }
}

Responder

Resonder 負責發送 RPC 請求結果給 Client,Scheduler 調度請求后,執行結果通過 doRespond() 加入到返回結果的相應隊列里面

class Responder extends Thread {

    void doRespond(Call call) throws IOException {
        boolean added = false;
        
        // 如果已經有一個正在進行的寫入,不會等待。
        // 這允許立即釋放處理程序以執行其他任務。
        if (call.connection.responseQueue.isEmpty() && 
            call.connection.responseWriteLock.tryLock()) {
            
            try {
                if (call.connection.responseQueue.isEmpty()) {
                    
                    // 這里如果完成寫操作,直接返回
                    if (processResponse(call)) {
                        return; // we're done.
                    }
                    
                    call.connection.responseQueue.addFirst(call);
                    added = true;
                }
            } finally {
                call.connection.responseWriteLock.unlock();
            }
        }

        if (!added) {
            call.connection.responseQueue.addLast(call);
        }
        
        // Add a connection to the list that want to write,
        call.responder.registerForWrite(call.connection);
    }
}

如果在 doRespond() 中沒有完成寫操作,通過將 Call 對象的 connection 注冊到 selector,由 Responder 中的線程進行后續的操作。

protected class Responder extends Thread {
    private final Selector writeSelector;

    public void registerForWrite(Connection c) {
        if (writingCons.add(c)) {
            writeSelector.wakeup();
        }
    }
    
    private void doRunLoop() {
        while (running) {
            try {
                // 獲取要寫入的連接列表,并在 selector 中注冊
                registerWrites();
                
                // ...

                Set<SelectionKey> keys = writeSelector.selectedKeys();
                Iterator<SelectionKey> iter = keys.iterator();
                while (iter.hasNext()) {
                    SelectionKey key = iter.next();
                    iter.remove();
                    if (key.isValid() && key.isWritable()) {
                        // 異步寫
                        doAsyncWrite(key);
                    }
                }
            } catch (OutOfMemoryError e) {
                // return or sleep  
            }
        }
    }

    private void doAsyncWrite(SelectionKey key) throws IOException {
        Connection connection = (Connection) key.attachment();
        if (processAllResponses(connection)) {
            // ...
        }
    }

    private boolean processAllResponses(final Connection connection) throws IOException {

        // Only one writer on the channel for a connection at a time.
        connection.responseWriteLock.lock();
        try {
            for (int i = 0; i < 20; i++) {
                Call call = connection.responseQueue.pollFirst();
            
                if (!processResponse(call)) {
                    connection.responseQueue.addFirst(call);
                    return false;
                }
            }
        } finally {
            connection.responseWriteLock.unlock();
        }

        return connection.responseQueue.isEmpty();
    }
}

?著作權歸作者所有,轉載或內容合作請聯系作者
平臺聲明:文章內容(如有圖片或視頻亦包括在內)由作者上傳并發布,文章內容僅代表作者本人觀點,簡書系信息發布平臺,僅提供信息存儲服務。

推薦閱讀更多精彩內容