- 消息從客戶端產(chǎn)生后，發(fā)送到哪個(gè)broker；
- 發(fā)送到broker后，broker如何接收，如何存儲(chǔ);
KafkaApis中響應(yīng)LeaderAndIsr Request
- 針對(duì)topic，KafkaController會(huì)進(jìn)行partition選主，產(chǎn)生最新的isr, replicas, 然后發(fā)送到topic的各個(gè)replica，詳細(xì)參考Kafka集群建立過(guò)程分析和KafkaController分析5-Partition狀態(tài)機(jī)
-
KafkaApis中處理LeaderAndIsr Request并發(fā)送LeaderAndIsrResponse；
val correlationId = request.header.correlationId
val leaderAndIsrRequest = request.body.asInstanceOf[LeaderAndIsrRequest]
def onLeadershipChange(updatedLeaders: Iterable[Partition], updatedFollowers: Iterable[Partition]) {
// for each new leader or follower, call coordinator to handle consumer group migration.
// this callback is invoked under the replica state change lock to ensure proper order of
// leadership changes
updatedLeaders.foreach { partition =>
if (partition.topic == GroupCoordinator.GroupMetadataTopicName)
coordinator.handleGroupImmigration(partition.partitionId)
}
updatedFollowers.foreach { partition =>
if (partition.topic == GroupCoordinator.GroupMetadataTopicName)
coordinator.handleGroupEmigration(partition.partitionId)
}
}
val responseHeader = new ResponseHeader(correlationId)
val leaderAndIsrResponse=
if (authorize(request.session, ClusterAction, Resource.ClusterResource)) {
val result = replicaManager.becomeLeaderOrFollower(correlationId, leaderAndIsrRequest, metadataCache, onLeadershipChange)
new LeaderAndIsrResponse(result.errorCode, result.responseMap.mapValues(new JShort(_)).asJava)
} else {
val result = leaderAndIsrRequest.partitionStates.asScala.keys.map((_, new JShort(Errors.CLUSTER_AUTHORIZATION_FAILED.code))).toMap
new LeaderAndIsrResponse(Errors.CLUSTER_AUTHORIZATION_FAILED.code, result.asJava)
}
requestChannel.sendResponse(new Response(request, new ResponseSend(request.connectionId, responseHeader, leaderAndIsrResponse)))
其中最主要的操作調(diào)用ReplicaManager.becomeLeaderOrFollower來(lái)初始化Partition
val result = replicaManager.becomeLeaderOrFollower(correlationId, leaderAndIsrRequest, metadataCache, onLeadershipChange)
ReplicaManager.becomeLeaderOrFollower
- 判斷
LeaderAndIsr請(qǐng)求中的controllerEpoch和ReplicaManager保存的controllerEpoch(在處理UpdateMetadata Request時(shí)更新, 參見(jiàn)Kafka集群Metadata管理), 如果本地存的controllerEpoch大，則忽略當(dāng)前的LeaderAndIsr請(qǐng)求, 產(chǎn)生BecomeLeaderOrFollowerResult(responseMap, ErrorMapping.StaleControllerEpochCode)； - 處理
leaderAndISRRequest.partitionStates中的第個(gè)partition state;
2.1 創(chuàng)建Partition對(duì)象，這個(gè)我們后面會(huì)講到;
allPartitions.putIfNotExists((topic, partitionId), new Partition(topic, partitionId, time, this))
2.2 如果partitionStateInfo中的leaderEpoch更新，則存儲(chǔ)它在val partitionState = new mutable.HashMap[Partition, LeaderAndIsrRequest.PartitionState]()中
if (partitionLeaderEpoch < stateInfo.leaderEpoch) {
if(stateInfo.replicas.contains(config.brokerId))
partitionState.put(partition, stateInfo)
}
2.3 分離出轉(zhuǎn)換成leader和follower的partitions;
val partitionsTobeLeader = partitionState.filter { case (partition, stateInfo) =>
stateInfo.leader == config.brokerId
}
val partitionsToBeFollower = (partitionState -- partitionsTobeLeader.keys)
2.4 處理轉(zhuǎn)換成leader
makeLeaders(controllerId, controllerEpoch, partitionsTobeLeader, correlationId, responseMap)
實(shí)現(xiàn)上干兩件事:
停止從leader來(lái)同步消息: replicaFetcherManager.removeFetcherForPartitions(partitionState.keySet.map(new TopicAndPartition(_))),參見(jiàn)ReplicaManager源碼解析1-消息同步線程管理
調(diào)用Partition的makeLeader方法:partition.makeLeader(controllerId, partitionStateInfo, correlationId)來(lái)作leader的轉(zhuǎn)換
2.5 處理轉(zhuǎn)換成follower
makeFollowers(controllerId, controllerEpoch, partitionsToBeFollower, leaderAndISRRequest.correlationId, responseMap, metadataCache)
2.6 啟動(dòng)HighWaterMarkCheckPointThread, 具體后面章節(jié)會(huì)講到,
if (!hwThreadInitialized) {
startHighWaterMarksCheckPointThread()
hwThreadInitialized = true
}
2.7 回調(diào)KafkaApis.handleLeaderAndIsrRequest.onLeadershipChange
- 針對(duì)
makeLeaders和makeFollowers的分析我們等分析完Parition,ReplicaFetcherManager后一并分析. - LeaderAndIsr 請(qǐng)求響應(yīng)流程圖:
LeaderAndIsr 請(qǐng)求響應(yīng).png
