RocketMQ源碼（一）：NameServer的啟動
RocketMQ源碼（二）：broker的啟動（一）
RocketMQ源碼（三）：broker的啟動（二）
RocketMQ源碼（四）：producer的啟動
RocketMQ源碼（五）：producer發送消息
RocketMQ源碼（六）：broker接收消息
RocketMQ源碼（八）：consumer消息拉取（一）
RocketMQ源碼（九）：consumer消息拉取（二）

consumer

• DefaultMQPullConsumer（deprecated）
• DefaultLitePullConsumer
• DefaultMQPushConsumer
  前兩種是通過pull的方式去拉取消息，第三種是push的方式。但是，其本質都是通過pull的方式拉取消息，也就是客戶端去broker獲取消息。
  因為push的模式下雖然消息的實時性比較好，但是如果要考慮到客戶端的消費速度慢于broker的投遞速度導致消息在內存中積壓，就會使程序變得很復雜。因此一般情況下都是使用pull模式來消費數據。
  DefaultMQPullConsumer在源碼中已經被標注為Deprecated，根據注釋看到DefaultMQPullConsumer將在2022年移除，因此這里主要是分析DefaultLitePullConsumer的工作原理。
  DefaultLitePullConsumer相比DefaultMQPullConsumer整體上要簡便一點，但是缺少了消息重試機制，只能通過seek()方法重置consumerOffset來實現消息的重復消費，而DefaultMQPullConsumer有sendMessageBack。litePull相比push模式以及老的pull方法比較大的一個區別是一個消費者默認可以用20個線程去拉取消息。

public class LitePullConsumerSubscribe {

    public static volatile boolean running = true;

    public static void main(String[] args) throws Exception {
        DefaultLitePullConsumer litePullConsumer = new DefaultLitePullConsumer("lite_pull_consumer_test");
        litePullConsumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
        litePullConsumer.subscribe("TopicTest", "*");
        litePullConsumer.setNamesrvAddr("127.0.0.1:9876");
        litePullConsumer.start();
        try {
            while (running) {
                List<MessageExt> messageExts = litePullConsumer.poll();
                System.out.printf("%s%n", messageExts);
            }
        } finally {
            litePullConsumer.shutdown();
        }
    }
}

從org.apache.rocketmq.example.simple.LitePullConsumerSubscribe可以看到如果使用DefaultLitePullConsumer進行消息的消費，接下來就對整個過程中使用到的一些方法進行分析。
首先來認識一下DefaultLitePullConsumer

private final DefaultLitePullConsumerImpl defaultLitePullConsumerImpl;

// 消息消費組
private String consumerGroup;

// 長輪詢模式下，consumer請求在broker端最長掛起時間
private long brokerSuspendMaxTimeMillis = 1000 * 20;

// 消息消費者拉取消息最大的超時時間，必須大于 brokerSuspendMaxTimeMillis
private long consumerTimeoutMillisWhenSuspend = 1000 * 30;

// 客戶端與 Broker 建立網絡連接的最大超時時間
private long consumerPullTimeoutMillis = 1000 * 10;

// 消息組消費模型：集群模式或者廣播模式
private MessageModel messageModel = MessageModel.CLUSTERING;

// 消息消費負載隊列變更事件
private MessageQueueListener messageQueueListener;

// 消息消費進度存儲器
private OffsetStore offsetStore;

// 消息消費隊列負載策略
private AllocateMessageQueueStrategy allocateMessageQueueStrategy = new AllocateMessageQueueAveragely();

private boolean unitMode = false;

// 設置是否提交消息消費進度
private boolean autoCommit = true;

// 每一個消費者拉取消息的線程數，相比于push模式很大的區別點
private int pullThreadNums = 20;

private static final long MIN_AUTOCOMMIT_INTERVAL_MILLIS = 1000;

// 自動匯報消息位點的間隔時間
private long autoCommitIntervalMillis = 5 * 1000;

// 一次消息拉取最多返回的消息條數
private int pullBatchSize = 10;

// 單個隊列積壓的消息條數觸發限流的闊值
private long pullThresholdForAll = 10000;

// 單個消息處理隊列中最大消息偏移量與最小偏移量的差值觸發限流的闊值
private int consumeMaxSpan = 2000;

// 單個隊列積壓的消息總大小觸發限流的闊值
private int pullThresholdForQueue = 1000;

// 單個隊列擠壓的消息總大小觸發限流的闊值
private int pullThresholdSizeForQueue = 100;

// 一次消息拉取默認超時時間
private long pollTimeoutMillis = 1000 * 5;

// topic 路由信息更新頻率
private long topicMetadataCheckIntervalMillis = 30 * 1000;

// 初次啟動時從什么位置開始消費
private ConsumeFromWhere consumeFromWhere = ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET;

// 如果初次啟動時 consumeFromWhere 策略選擇為基于時間戳，通過該屬性設置定位的時間
private String consumeTimestamp = UtilAll.timeMillisToHumanString3(System.currentTimeMillis() - (1000 * 60 * 30));

可以看到的是DefaultLitePullConsumer基本是設置了一些屬性，具體的功能實現都是調用的DefaultLitePullConsumerImpl，接下來看看它的一些主要方法

// 啟動consumer
@Override
public void start() throws MQClientException {
    setConsumerGroup(NamespaceUtil.wrapNamespace(this.getNamespace(), this.consumerGroup));
    this.defaultLitePullConsumerImpl.start();
}
// 關閉consumer
@Override
public void shutdown() {
    this.defaultLitePullConsumerImpl.shutdown();
}
// 使用subExpression表達式訂閱topic，根據負載均衡算法計算分配的隊列
@Override
public void subscribe(String topic, String subExpression) throws MQClientException {
    this.defaultLitePullConsumerImpl.subscribe(withNamespace(topic), subExpression);
}
// 強制分配消費隊列，跳過負載均衡算法
@Override
public void assign(Collection<MessageQueue> messageQueues) {
    defaultLitePullConsumerImpl.assign(queuesWithNamespace(messageQueues));
}
// 從內存中獲取消息
@Override
public List<MessageExt> poll() {
    return defaultLitePullConsumerImpl.poll(this.getPollTimeoutMillis());
}
// 重置消費offset
@Override
public void seek(MessageQueue messageQueue, long offset) throws MQClientException {
    this.defaultLitePullConsumerImpl.seek(queueWithNamespace(messageQueue), offset);
}
// 停止從broker拉取對應MessageQueue的消息
@Override
public void pause(Collection<MessageQueue> messageQueues) {
    this.defaultLitePullConsumerImpl.pause(queuesWithNamespace(messageQueues));
}
// 從NameServer獲取topic下所有的messageQueue
@Override
public Collection<MessageQueue> fetchMessageQueues(String topic) throws MQClientException {
    return this.defaultLitePullConsumerImpl.fetchMessageQueues(withNamespace(topic));
}
// 監聽topic的messageQueue變化事件
@Override
public void registerTopicMessageQueueChangeListener(String topic,
    TopicMessageQueueChangeListener topicMessageQueueChangeListener) throws MQClientException {
    this.defaultLitePullConsumerImpl.registerTopicMessageQueueChangeListener(withNamespace(topic), topicMessageQueueChangeListener);
}
// 異步提交所有隊列的offset
@Override
public void commitSync() {
    this.defaultLitePullConsumerImpl.commitAll();
}
// 設置是否自動提交offset
@Override
public void setAutoCommit(boolean autoCommit) {
    this.autoCommit = autoCommit;
}

對DefaultLitePullConsumer有了基本認識后，還是根據LitePullConsumerSubscribe這個demo類來分析DefaultLitePullConsumer的一些常規操作流程
這里主要分成兩部分來看：

• consumer的啟動
• 消息的拉取
  這里先看看consumer的啟動過程，首先看看DefaultLitePullConsumer的實例化方法。

public DefaultLitePullConsumer(final String consumerGroup) {
    this(null, consumerGroup, null);
}

public DefaultLitePullConsumer(final String namespace, final String consumerGroup, RPCHook rpcHook) {
    this.namespace = namespace;
    this.consumerGroup = consumerGroup;
    defaultLitePullConsumerImpl = new DefaultLitePullConsumerImpl(this, rpcHook);
}

public DefaultLitePullConsumerImpl(final DefaultLitePullConsumer defaultLitePullConsumer, final RPCHook rpcHook) {
    this.defaultLitePullConsumer = defaultLitePullConsumer;
    this.rpcHook = rpcHook;
    // 消息拉取線程池
    this.scheduledThreadPoolExecutor = new ScheduledThreadPoolExecutor(
        this.defaultLitePullConsumer.getPullThreadNums(),
        new ThreadFactoryImpl("PullMsgThread-" + this.defaultLitePullConsumer.getConsumerGroup())
    );
    // 監視topic的messageQueue變化線程
    this.scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(new ThreadFactory() {
        @Override
        public Thread newThread(Runnable r) {
            return new Thread(r, "MonitorMessageQueueChangeThread");
        }
    });
    // 發生異常時拉取消息的延遲時間
    this.pullTimeDelayMillsWhenException = defaultLitePullConsumer.getPullTimeDelayMillsWhenException();
}

接下來是調用DefaultLitePullConsumer的subscribe方法，訂閱某個topic的消息

public synchronized void subscribe(String topic, String subExpression) throws MQClientException {
    try {
        if (topic == null || topic.equals("")) {
            throw new IllegalArgumentException("Topic can not be null or empty.");
        }
        // 設置訂閱類型
        setSubscriptionType(SubscriptionType.SUBSCRIBE);
        // 創建訂閱信息
        SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(defaultLitePullConsumer.getConsumerGroup(),
            topic, subExpression);
        // 保存訂閱關系
        this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
        // 注冊 messageQueue 的變化監聽，觸發消息拉取
        this.defaultLitePullConsumer.setMessageQueueListener(new MessageQueueListenerImpl());
        assignedMessageQueue.setRebalanceImpl(this.rebalanceImpl);
        if (serviceState == ServiceState.RUNNING) {
            this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
            // 再次確認是否更新訂閱信息
            updateTopicSubscribeInfoWhenSubscriptionChanged();
        }
    } catch (Exception e) {
        throw new MQClientException("subscribe exception", e);
    }
}

SubscriptionType一共有三種

• NONE 未知
• SUBSCRIBE 訂閱，負載均衡算法分配消費的隊列
• ASSIGN 分配，指定消費隊列，跳過負載均衡算法
  首先看下是怎么創建訂閱信息的

public static SubscriptionData buildSubscriptionData(final String consumerGroup, String topic,
    String subString) throws Exception {
    // 創建SubscriptionData，保存訂閱信息
    SubscriptionData subscriptionData = new SubscriptionData();
    subscriptionData.setTopic(topic);
    subscriptionData.setSubString(subString);
    // 處理訂閱表達式
    if (null == subString || subString.equals(SubscriptionData.SUB_ALL) || subString.length() == 0) {
        subscriptionData.setSubString(SubscriptionData.SUB_ALL);
    } else {
        // 根據 || 分割訂閱的多個tag
        String[] tags = subString.split("\\|\\|");
        if (tags.length > 0) {
            for (String tag : tags) {
                if (tag.length() > 0) {
                    String trimString = tag.trim();
                    // 保存tag和tag的hashcode
                    if (trimString.length() > 0) {
                        subscriptionData.getTagsSet().add(trimString);
                        subscriptionData.getCodeSet().add(trimString.hashCode());
                    }
                }
            }
        } else {
            throw new Exception("subString split error");
        }
    }

    return subscriptionData;
}

方法比較簡單
回到subscribe中，這里定義了一個MessageQueueListenerImpl，這個類就是消息拉取的具體實現，后續會用到
這里還有個判斷邏輯是serviceState == ServiceState.RUNNING是，代表著程序先調用了DefaultLitePullConsumer的start方法，然后才調用了subscribe，這個時候就需要向broker發送心跳，并且更新訂閱關系。
接下來看看DefaultLitePullConsumer的start方法

public void start() throws MQClientException {
    setConsumerGroup(NamespaceUtil.wrapNamespace(this.getNamespace(), this.consumerGroup));
    this.defaultLitePullConsumerImpl.start();
}

public synchronized void start() throws MQClientException {
    switch (this.serviceState) {
        case CREATE_JUST:
            // 和producer啟動一樣，先設置狀態為 啟動失敗
            this.serviceState = ServiceState.START_FAILED;
            // 校驗一些基礎配置
            this.checkConfig();
            // 如果是集群模式且instanceName=DEFAULT，修改instanceName為pid
            if (this.defaultLitePullConsumer.getMessageModel() == MessageModel.CLUSTERING) {
                this.defaultLitePullConsumer.changeInstanceNameToPID();
            }
            // 初始化 MQClientFactory
            initMQClientFactory();
            // 初始化 RebalanceImpl，主要是屬性設置
            initRebalanceImpl();
            // 實例化 PullAPIWrapper
            initPullAPIWrapper();
            // 實例化 offsetStore
            initOffsetStore();
            // 同 producer 的啟動過程
            mQClientFactory.start();
            // 從 NameServer 獲取 messageQueue，和本地的比較
            // 如果變化則通知 TopicMessageQueueChangeListener
            startScheduleTask();

            this.serviceState = ServiceState.RUNNING;

            log.info("the consumer [{}] start OK", this.defaultLitePullConsumer.getConsumerGroup());
            // 用于處理 subscribe 或 assign 方法在 start 前就已經執行
            // 并且保存 topic 的 messageQueue 信息到 messageQueuesForTopic
            operateAfterRunning();

            break;
        case RUNNING:
        case START_FAILED:
        case SHUTDOWN_ALREADY:
            throw new MQClientException("The PullConsumer service state not OK, maybe started once, "
                + this.serviceState
                + FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
                null);
        default:
            break;
    }
}

這個過程中初始化了一些屬性，initMQClientFactory和producer啟動中的一樣，initPullAPIWrapper是用于注冊鉤子方法，因此繼續向下看initRebalanceImpl方法

private void initRebalanceImpl() {
    this.rebalanceImpl.setConsumerGroup(this.defaultLitePullConsumer.getConsumerGroup());
    this.rebalanceImpl.setMessageModel(this.defaultLitePullConsumer.getMessageModel());
    this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultLitePullConsumer.getAllocateMessageQueueStrategy());
    this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);
}

private void initOffsetStore() throws MQClientException {
    if (this.defaultLitePullConsumer.getOffsetStore() != null) {
        this.offsetStore = this.defaultLitePullConsumer.getOffsetStore();
    } else {
        switch (this.defaultLitePullConsumer.getMessageModel()) {
            case BROADCASTING:
                this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultLitePullConsumer.getConsumerGroup());
                break;
            case CLUSTERING:
                this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultLitePullConsumer.getConsumerGroup());
                break;
            default:
                break;
        }
        this.defaultLitePullConsumer.setOffsetStore(this.offsetStore);
    }
    // 用于 LocalFileOffsetStore 從本地磁盤加載消費進度
    this.offsetStore.load();
}

初始化了一個rebalanceImpl和offsetStore，接下來的重頭戲是mQClientFactory的start方法

public void start() throws MQClientException {
    synchronized (this) {
        switch (this.serviceState) {
            case CREATE_JUST:
                this.serviceState = ServiceState.START_FAILED;
                // If not specified,looking address from name server
                if (null == this.clientConfig.getNamesrvAddr()) {
                    this.mQClientAPIImpl.fetchNameServerAddr();
                }
                // 開啟 netty 服務
                this.mQClientAPIImpl.start();
                // 開啟一些定時任務
                this.startScheduledTask();
                // 開啟拉取消息服務，PUSH模式的消費者使用，pull模式下線程會一直等待
                this.pullMessageService.start();
                // consumer平衡消費組和消息隊列的關系
                this.rebalanceService.start();
                // Start push service
                this.defaultMQProducer.getDefaultMQProducerImpl().start(false);
                log.info("the client factory [{}] start OK", this.clientId);
                this.serviceState = ServiceState.RUNNING;
                break;
            case START_FAILED:
                throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null);
            default:
                break;
        }
    }
}

startScheduledTask開啟的一批定時任務中有一個定時器是

// 持久化 consumer 的消費進度
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

    @Override
    public void run() {
        try {
            MQClientInstance.this.persistAllConsumerOffset();
        } catch (Exception e) {
            log.error("ScheduledTask persistAllConsumerOffset exception", e);
        }
    }
}, 1000 * 10, this.clientConfig.getPersistConsumerOffsetInterval(), TimeUnit.MILLISECONDS);

繼續分析persistAllConsumerOffset方法

private void persistAllConsumerOffset() {
    // 獲取所有的消費者
    Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
    while (it.hasNext()) {
        Entry<String, MQConsumerInner> entry = it.next();
        MQConsumerInner impl = entry.getValue();
        impl.persistConsumerOffset();
    }
}

這里主要看下DefaultLitePullConsumerImpl對persistConsumerOffset的實現

public void persistConsumerOffset() {
    try {
        checkServiceState();
        Set<MessageQueue> mqs = new HashSet<MessageQueue>();
        // 判斷當前 consumer 的訂閱類型
        // 如果是訂閱就取處理中的隊列
        if (this.subscriptionType == SubscriptionType.SUBSCRIBE) {
            Set<MessageQueue> allocateMq = this.rebalanceImpl.getProcessQueueTable().keySet();
            mqs.addAll(allocateMq);
        }
        // 如果是分配的
        else if (this.subscriptionType == SubscriptionType.ASSIGN) {
            Set<MessageQueue> assignedMessageQueue = this.assignedMessageQueue.getAssignedMessageQueues();
            mqs.addAll(assignedMessageQueue);
        }
        // 持久化隊列消費 offset 到遠端或者是本地
        this.offsetStore.persistAll(mqs);
    } catch (Exception e) {
        log.error("Persist consumer offset error for group: {} ", this.defaultLitePullConsumer.getConsumerGroup(), e);
    }
}

根據subscriptionType獲取到對應的MessageQueue集合后，調用offsetStore.persistAll
集群模式下offsetStore就是RemoteBrokerOffsetStore，所以繼續看RemoteBrokerOffsetStore對persistAll的實現

public void persistAll(Set<MessageQueue> mqs) {
    if (null == mqs || mqs.isEmpty())
        return;
    // 持久化消息隊列
    final HashSet<MessageQueue> unusedMQ = new HashSet<MessageQueue>();
    // offsetTable中的數據來源是當DefaultLitePullConsumer設置autoCommit為true是，在poll方法中會自動更新offsetTable中的offset
    for (Map.Entry<MessageQueue, AtomicLong> entry : this.offsetTable.entrySet()) {
        MessageQueue mq = entry.getKey();
        AtomicLong offset = entry.getValue();
        if (offset != null) {
            if (mqs.contains(mq)) {
                try {
                    // 把offset更新到broker去
                    this.updateConsumeOffsetToBroker(mq, offset.get());
                    log.info("[persistAll] Group: {} ClientId: {} updateConsumeOffsetToBroker {} {}",
                        this.groupName,
                        this.mQClientFactory.getClientId(),
                        mq,
                        offset.get());
                } catch (Exception e) {
                    log.error("updateConsumeOffsetToBroker exception, " + mq.toString(), e);
                }
            } else {
                unusedMQ.add(mq);
            }
        }
    }
    // 移除不適用的消息隊列
    if (!unusedMQ.isEmpty()) {
        for (MessageQueue mq : unusedMQ) {
            this.offsetTable.remove(mq);
            log.info("remove unused mq, {}, {}", mq, this.groupName);
        }
    }
}

這里就是自動提交Offset的相關實現，繼續看是如何實現的

private void updateConsumeOffsetToBroker(MessageQueue mq, long offset) throws RemotingException,
    MQBrokerException, InterruptedException, MQClientException {
    updateConsumeOffsetToBroker(mq, offset, true);
}

public void updateConsumeOffsetToBroker(MessageQueue mq, long offset, boolean isOneway) throws RemotingException,
    MQBrokerException, InterruptedException, MQClientException {
    // 查詢Broker地址信息
    FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInAdmin(mq.getBrokerName());
    // 如果沒有查詢到，就從NameServer獲取topic的路由信息
    if (null == findBrokerResult) {
        this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
        findBrokerResult = this.mQClientFactory.findBrokerAddressInAdmin(mq.getBrokerName());
    }

    if (findBrokerResult != null) {
        UpdateConsumerOffsetRequestHeader requestHeader = new UpdateConsumerOffsetRequestHeader();
        requestHeader.setTopic(mq.getTopic());
        requestHeader.setConsumerGroup(this.groupName);
        requestHeader.setQueueId(mq.getQueueId());
        requestHeader.setCommitOffset(offset);

        if (isOneway) {
            this.mQClientFactory.getMQClientAPIImpl().updateConsumerOffsetOneway(
                findBrokerResult.getBrokerAddr(), requestHeader, 1000 * 5);
        } else {
            this.mQClientFactory.getMQClientAPIImpl().updateConsumerOffset(
                findBrokerResult.getBrokerAddr(), requestHeader, 1000 * 5);
        }
    } else {
        throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
    }
}

public void updateConsumerOffsetOneway(
    final String addr,
    final UpdateConsumerOffsetRequestHeader requestHeader,
    final long timeoutMillis
) throws RemotingConnectException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException,
    InterruptedException {
    RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.UPDATE_CONSUMER_OFFSET, requestHeader);

    this.remotingClient.invokeOneway(MixAll.brokerVIPChannel(this.clientConfig.isVipChannelEnabled(), addr), request, timeoutMillis);
}

可以看到最終是向broker發送了一個RequestCode=UPDATE_CONSUMER_OFFSET的請求
回到MQClientInstance的start方法中，接下來是pullMessageService.start()和push模式的消息拉取相關，暫不分析
然后是rebalanceService.start()

public void run() {
    log.info(this.getServiceName() + " service started");

    while (!this.isStopped()) {
        // 默認間隔20s執行一次
        this.waitForRunning(waitInterval);
        this.mqClientFactory.doRebalance();
    }

    log.info(this.getServiceName() + " service end");
}

// 循環遍歷每個消費組獲取 MQConsumeInner 對象
// 并執行其 doRebalance 方法
public void doRebalance() {
    for (Map.Entry<String, MQConsumerInner> entry : this.consumerTable.entrySet()) {
        MQConsumerInner impl = entry.getValue();
        if (impl != null) {
            try {
                impl.doRebalance();
            } catch (Throwable e) {
                log.error("doRebalance exception", e);
            }
        }
    }
}

接下來繼續看DefaultLitePullConsumerImpl的doRebalance實現

public void doRebalance() {
    if (this.rebalanceImpl != null) {
        this.rebalanceImpl.doRebalance(false);
    }
}

// isOrder這里是false
public void doRebalance(final boolean isOrder) {
    // 獲取topic的訂閱關系
    // 如果沒有執行DefaultLitePullConsumer的subscribe方法，也就不會有訂閱關系，也就不會負載均衡
    Map<String, SubscriptionData> subTable = this.getSubscriptionInner();
    if (subTable != null) {
        for (final Map.Entry<String, SubscriptionData> entry : subTable.entrySet()) {
            final String topic = entry.getKey();
            try {
                // 對每一個topic進行負載均衡
                this.rebalanceByTopic(topic, isOrder);
            } catch (Throwable e) {
                if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                    log.warn("rebalanceByTopic Exception", e);
                }
            }
        }
    }
    // 移除 MessageQueue，如果 MesageQueue 的 topic 不在訂閱的主題中
    this.truncateMessageQueueNotMyTopic();
}

接下來繼續看負載均衡的具體實現

private void rebalanceByTopic(final String topic, final boolean isOrder) {
    switch (messageModel) {
        // 廣播模式
        case BROADCASTING: {
            Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
            if (mqSet != null) {
                boolean changed = this.updateProcessQueueTableInRebalance(topic, mqSet, isOrder);
                if (changed) {
                    this.messageQueueChanged(topic, mqSet, mqSet);
                    log.info("messageQueueChanged {} {} {} {}",
                        consumerGroup,
                        topic,
                        mqSet,
                        mqSet);
                }
            } else {
                log.warn("doRebalance, {}, but the topic[{}] not exist.", consumerGroup, topic);
            }
            break;
        }
        // 集群模式
        case CLUSTERING: {
            // 主題的消息消費隊列
            Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
            // 主題的當前消費組的消費者id列表
            List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
            if (null == mqSet) {
                if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                    log.warn("doRebalance, {}, but the topic[{}] not exist.", consumerGroup, topic);
                }
            }

            if (null == cidAll) {
                log.warn("doRebalance, {} {}, get consumer id list failed", consumerGroup, topic);
            }
            // 主要是對主題的消息隊列排序、消費者ID進行排序，然后利用分配算法，計算當前消費者ID(mqClient.clientId) 分配出需要拉取的消息隊列
            if (mqSet != null && cidAll != null) {
                List<MessageQueue> mqAll = new ArrayList<MessageQueue>();
                mqAll.addAll(mqSet);

                Collections.sort(mqAll);
                Collections.sort(cidAll);

                AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;
                // 根據 隊列分配策略 分配消息隊列
                List<MessageQueue> allocateResult = null;
                try {
                    allocateResult = strategy.allocate(
                        this.consumerGroup,
                        this.mQClientFactory.getClientId(),
                        mqAll,
                        cidAll);
                } catch (Throwable e) {
                    log.error("AllocateMessageQueueStrategy.allocate Exception. allocateMessageQueueStrategyName={}", strategy.getName(),
                        e);
                    return;
                }

                Set<MessageQueue> allocateResultSet = new HashSet<MessageQueue>();
                if (allocateResult != null) {
                    allocateResultSet.addAll(allocateResult);
                }
                // 更新主題的消息消費處理隊列，并返回消息隊列負載是否改變
                boolean changed = this.updateProcessQueueTableInRebalance(topic, allocateResultSet, isOrder);
                if (changed) {
                    log.info(
                        "rebalanced result changed. allocateMessageQueueStrategyName={}, group={}, topic={}, clientId={}, mqAllSize={}, cidAllSize={}, rebalanceResultSize={}, rebalanceResultSet={}",
                        strategy.getName(), consumerGroup, topic, this.mQClientFactory.getClientId(), mqSet.size(), cidAll.size(),
                        allocateResultSet.size(), allocateResultSet);
                    // 通知 MessageQueueListener ，分配的隊列信息變更
                    this.messageQueueChanged(topic, mqSet, allocateResultSet);
                }
            }
            break;
        }
        default:
            break;
    }
}

這里主要關注集群模式下的處理方式，主要做了三件事

• 根據負載均衡算法，獲取分配到的隊列
• 判斷負責消費的隊列信息是否變更
• 發布消費的隊列信息變更事件
  負載均衡算法具體不分析，接下來看看updateProcessQueueTableInRebalance

private boolean updateProcessQueueTableInRebalance(final String topic, final Set<MessageQueue> mqSet,
    final boolean isOrder) {
    boolean changed = false;
    // 遍歷消息隊列-處理隊列緩存
    Iterator<Entry<MessageQueue, ProcessQueue>> it = this.processQueueTable.entrySet().iterator();
    while (it.hasNext()) {
        Entry<MessageQueue, ProcessQueue> next = it.next();
        MessageQueue mq = next.getKey();
        ProcessQueue pq = next.getValue();
        // 只處理 mq 的主題與該主題相關的 ProcessQueue
        if (mq.getTopic().equals(topic)) {
            // 如果 mq 不在當期主題的處理范圍內
            if (!mqSet.contains(mq)) {
                // 首先設置該消息隊列為丟棄
                pq.setDropped(true);
                // 判斷是否需要移除
                if (this.removeUnnecessaryMessageQueue(mq, pq)) {
                    it.remove();
                    changed = true;
                    log.info("doRebalance, {}, remove unnecessary mq, {}", consumerGroup, mq);
                }
            }
            // 距離上次拉取，超過最長等待時間
            else if (pq.isPullExpired()) {
                switch (this.consumeType()) {
                    // 如果是 pull 模式
                    case CONSUME_ACTIVELY:
                        break;
                    // 如果是 push 模式
                    case CONSUME_PASSIVELY:
                        pq.setDropped(true);
                        if (this.removeUnnecessaryMessageQueue(mq, pq)) {
                            it.remove();
                            changed = true;
                            log.error("[BUG]doRebalance, {}, remove unnecessary mq, {}, because pull is pause, so try to fixed it",
                                consumerGroup, mq);
                        }
                        break;
                    default:
                        break;
                }
            }
        }
    }
    // 增加 不在processQueueTable && 存在于mqSet 里的消息隊列。
    List<PullRequest> pullRequestList = new ArrayList<PullRequest>();
    for (MessageQueue mq : mqSet) {
        // 處理新分配過來的隊列
        if (!this.processQueueTable.containsKey(mq)) {
            if (isOrder && !this.lock(mq)) {
                log.warn("doRebalance, {}, add a new mq failed, {}, because lock failed", consumerGroup, mq);
                continue;
            }
            // 在內存中移除 MessageQueue 的 offerset
            this.removeDirtyOffset(mq);
            ProcessQueue pq = new ProcessQueue();
            // 計算下一個拉取偏移量
            long nextOffset = this.computePullFromWhere(mq);
            if (nextOffset >= 0) {
                // 創建一個拉取任務
                ProcessQueue pre = this.processQueueTable.putIfAbsent(mq, pq);
                if (pre != null) {
                    log.info("doRebalance, {}, mq already exists, {}", consumerGroup, mq);
                } else {
                    log.info("doRebalance, {}, add a new mq, {}", consumerGroup, mq);
                    PullRequest pullRequest = new PullRequest();
                    pullRequest.setConsumerGroup(consumerGroup);
                    pullRequest.setNextOffset(nextOffset);
                    pullRequest.setMessageQueue(mq);
                    pullRequest.setProcessQueue(pq);
                    pullRequestList.add(pullRequest);
                    changed = true;
                }
            } else {
                log.warn("doRebalance, {}, add new mq failed, {}", consumerGroup, mq);
            }
        }
    }
    // 立刻執行拉取消息請求
    this.dispatchPullRequest(pullRequestList);

    return changed;
}

值得一提的是computePullFromWhere方法，看下這里是如何得知下一次消費開始的offset

public long computePullFromWhere(MessageQueue mq) {
    // 獲取consumeFromWhere配置
    ConsumeFromWhere consumeFromWhere = litePullConsumerImpl.getDefaultLitePullConsumer().getConsumeFromWhere();
    long result = -1;
    switch (consumeFromWhere) {
        case CONSUME_FROM_LAST_OFFSET: {
            long lastOffset = litePullConsumerImpl.getOffsetStore().readOffset(mq, ReadOffsetType.MEMORY_FIRST_THEN_STORE);
            if (lastOffset >= 0) {
                result = lastOffset;
            } else if (-1 == lastOffset) {
                // 重試隊列
                if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) { // First start, no offset
                    result = 0L;
                }
                // 否則獲取最大的offset
                else {
                    try {
                        result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
                    } catch (MQClientException e) {
                        result = -1;
                    }
                }
            } else {
                result = -1;
            }
            break;
        }
        case CONSUME_FROM_FIRST_OFFSET: {
            long lastOffset = litePullConsumerImpl.getOffsetStore().readOffset(mq, ReadOffsetType.MEMORY_FIRST_THEN_STORE);
            if (lastOffset >= 0) {
                result = lastOffset;
            } else if (-1 == lastOffset) {
                result = 0L;
            } else {
                result = -1;
            }
            break;
        }
        case CONSUME_FROM_TIMESTAMP: {
            long lastOffset = litePullConsumerImpl.getOffsetStore().readOffset(mq, ReadOffsetType.MEMORY_FIRST_THEN_STORE);
            if (lastOffset >= 0) {
                result = lastOffset;
            } else if (-1 == lastOffset) {
                if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                    try {
                        result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
                    } catch (MQClientException e) {
                        result = -1;
                    }
                } else {
                    try {
                        long timestamp = UtilAll.parseDate(this.litePullConsumerImpl.getDefaultLitePullConsumer().getConsumeTimestamp(),
                            UtilAll.YYYYMMDDHHMMSS).getTime();
                        result = this.mQClientFactory.getMQAdminImpl().searchOffset(mq, timestamp);
                    } catch (MQClientException e) {
                        result = -1;
                    }
                }
            } else {
                result = -1;
            }
            break;
        }
    }
    return result;
}

public long readOffset(final MessageQueue mq, final ReadOffsetType type) {
    if (mq != null) {
        switch (type) {
            // 先從內存中讀取，如果內存中不存在，再嘗試從遠程中讀取
            case MEMORY_FIRST_THEN_STORE:
            // 從內存中讀取
            case READ_FROM_MEMORY: {
                AtomicLong offset = this.offsetTable.get(mq);
                if (offset != null) {
                    return offset.get();
                } else if (ReadOffsetType.READ_FROM_MEMORY == type) {
                    return -1;
                }
            }
            // 從遠程讀取然后更新到本地
            case READ_FROM_STORE: {
                try {
                    long brokerOffset = this.fetchConsumeOffsetFromBroker(mq);
                    AtomicLong offset = new AtomicLong(brokerOffset);
                    this.updateOffset(mq, offset.get(), false);
                    return brokerOffset;
                }
                // No offset in broker
                catch (MQBrokerException e) {
                    return -1;
                }
                //Other exceptions
                catch (Exception e) {
                    log.warn("fetchConsumeOffsetFromBroker exception, " + mq, e);
                    return -2;
                }
            }
            default:
                break;
        }
    }

    return -1;
}

可以看到不管是哪種consumeFromWhere配置，都是默認先從本地或broker獲取最新的offset消費進度。不一樣的是沒有獲取到消費進度時，每種consumeFromWhere的處理方式。
回到RebalanceImpl的rebalanceByTopic方法，繼續看RebalanceLitePullImpl的messageQueueChanged方法

public void messageQueueChanged(String topic, Set<MessageQueue> mqAll, Set<MessageQueue> mqDivided) {
    MessageQueueListener messageQueueListener = this.litePullConsumerImpl.getDefaultLitePullConsumer().getMessageQueueListener();
    if (messageQueueListener != null) {
        try {
            messageQueueListener.messageQueueChanged(topic, mqAll, mqDivided);
        } catch (Throwable e) {
            log.error("messageQueueChanged exception", e);
        }
    }
}

這里和消息拉取相關，后續分析
到此consumer的啟動過程就分析完了