Kafka不同進程“溝通機制”
Kafka服務的定位是一種高吞吐量的分布式消息訂閱系統(tǒng)。服務再運行過程中,不同的進程(broker和controller之間,客戶端和集群)之間需要進行“溝通”來保證功能的可用性。
Kafka主要通過兩種方式進行“溝通”,以保證自身狀態(tài)或需求被感知:
通過nio方式完成“溝通”
controller通過ControllerChannelManager發(fā)送消息到各broker,主要有ApiKeys.LEADER_AND_ISR, ApiKeys.STOP_REPLICA, ApiKeys.UPDATE_METADATA_KEY 三種消息。其中僅有ApiKeys.STOP_REPLICA有相關callback操作。broker通過KafkaRequestHandlerPool接收消息并調用kafkaApi進行消息相應。其他如producer發(fā)送消息,consumser消費消息,獲取集群信息等操作,也是客戶端通過nio的方式發(fā)送請求。Broker可處理的消息有如下:
消息 | 備注 |
---|---|
ApiKeys.LEADER_AND_ISR | Controller 發(fā)送給broker |
ApiKeys.STOP_REPLICA | Controller 發(fā)送給broker |
ApiKeys.UPDATE_METADATA_KEY | Controller 發(fā)送給broker |
ApiKeys.PRODUCE | producer發(fā)送生產(chǎn)數(shù)據(jù)請求給broker |
ApiKeys.FETCH | consumer發(fā)送消費數(shù)據(jù)請求給broker |
ApiKeys.LIST_OFFSETS | 客戶端/consumer 發(fā)送查詢offset請求 |
ApiKeys.METADATA | 獲取集群topic信息 |
ApiKeys.CONTROLLED_SHUTDOWN_KEY | brokershutdown時向controller發(fā)送消息 |
ApiKeys.OFFSET_COMMIT | consumer向coordinator發(fā)送的offset commit 請求 |
ApiKeys.OFFSET_FETCH | consumer發(fā)送的請求partitions的當前的offset |
ApiKeys.GROUP_COORDINATOR | consumer向broker發(fā)送group的coordinator是哪個節(jié)點的請求 |
ApiKeys.JOIN_GROUP | consumer向groupcoordinator發(fā)送 join group的請求 |
ApiKeys.HEARTBEAT | consumer定期發(fā)送heartbeat請求到groupcoordinator |
ApiKeys.LEAVE_GROUP | consumer停止消費時發(fā)送退出group請求到groupcoordinator |
ApiKeys.SYNC_GROUP | consumer變?yōu)閘eader獲取follower時發(fā)送的已同步請求到groupcoordinator |
ApiKeys.DESCRIBE_GROUPS | 客戶端工具kafka-consumer-groups.sh ConsumerGroupCommand中使用describe group信息所用 |
ApiKeys.LIST_GROUPS | 客戶端工具kafka-consumer-groups.sh ConsumerGroupCommand中使用list group信息所用 |
ApiKeys.SASL_HANDSHAKE | nio的select發(fā)送消息/請求時,需要先發(fā)送sasl——handshake請求 |
ApiKeys.API_VERSIONS | 客戶端發(fā)送的kafka版本請求 |
通過zookeeper注冊監(jiān)聽器完成“溝通”
客戶端(broker/client)如果需要將訴求(如topic創(chuàng)建,刪除,reblance,partition reassign等)告知服務,所采取的方案是再ZooKeeper中創(chuàng)建目錄,寫入數(shù)據(jù),服務端(此處指controller)通過listener監(jiān)控到zk中的變化,并作出響應
Zookeeper的watch機制與第三方封裝
Zookeeper是分布式應用程序協(xié)調服務。不同的進程通過約定的目錄的寫入和監(jiān)控操作完成消息的互相感知。開源大數(shù)據(jù)多數(shù)服務的高可用性實現(xiàn)都是依靠zookeeper的協(xié)調功能完成。zookeeper客戶端通過watcher機制完成對zookeeper的集群的監(jiān)控。
由于zookeeper應用開發(fā)的復雜性,出現(xiàn)了第三方對zk的應用開發(fā)的封裝。常用的有Curator FrameWork,I0Itec-zkclient。其中hive,hbase,spark等都是通過Curator FrameWork完成和zk的交互。Kafka則選用的是I0Itec-zkclient與zk交互。
Kafka中實現(xiàn)listener的方法:
- Kafka通過zkUtils中調用org.I0Itec.zkclient.zkCLient完成listener(listener中相關的handle方法有相關的處理邏輯)的注冊。
- zkClient繼承Watcher,再監(jiān)控的對象發(fā)生變化時,zkCLient中的process方法會被調用
- 在該方法中會完成對不同的listener的handle方法的調用
I0Itec-zkclient主要有如下三種listener,分別實現(xiàn)不同的監(jiān)聽方式。
消息 | handle的方法 | 備注 |
---|---|---|
IZkChildListener | handleChildChange | listening on zk child changes for a given path |
IZkDataListener | handleDataChange | listening on zk data changes for a given path |
IZkDataListener | handleDataDeleted | listening on zk data changes for a given path |
IZkStateListener | handleStateChanged | Called when the zookeeper connection state has changed. |
IZkStateListener | handleNewSession | Called after the zookeeper session has expired and a new session has been created |
IZkStateListener | handleSessionEstablishmentError | Called when a session cannot be re-established |
listener接口 | listener | listener 模塊 | 監(jiān)控對象 | 備注 |
---|---|---|---|---|
IZkChildListener | DeleteTopicsListener | PartitionStateMachine | DeleteTopicsPath = "/admin/delete_topics" | |
IZkChildListener | IsrChangeNotificationListener | KafkaController | IsrChangeNotificationPath = "/isr_change_notification" | |
IZkChildListener | zkTopicEventListener | ZookeeperTOpicEventWatcher | BrokerTopicsPath = "/brokers/topics" | |
IZkChildListener | NodeChangeListener | ZkNodeCHangeNotificaionListener in SimpleAclAuthorizer | AclChangedZkPath = "/kafka-acl-changes" AclChangedPrefix = "acl_changes_" | |
IZkChildListener | NodeChangeListener | ZkNodeCHangeNotificaionListener In DynamicConfigManager | EntityConfigChangesPath = "/config/changes" EntityConfigChangeZnodePrefix = "config_change_" | |
IZkChildListener | ZKRebalancerListener | ZookeeperConsumerConnector | BrokerIdsPath = "/brokers/ids" | 同時也會被由ZKTopicPartitionChangeListener觸發(fā),然后調用syncedRebalance進行rebalance |
IZkChildListener | TopicChangeListener | PartitionStateMachine | BrokerTopicsPath = "/brokers/topics" | |
IZkChildListener | BrokerChangeListener | ReplicaStateMachine | BrokerIdsPath = "/brokers/ids" | |
IZKDateListener | PreferredReplicaElectionListener | KafkaController | PreferredReplicaLeaderElectionPath = "/admin/preferred_replica_election" | |
IZKDateListener | ZKTopicPartitionChangeListener | ZookeeperConsumerConnector | "/brokers/topics/${topic_name}" | consumer監(jiān)控的partition的變化并作出處理。 高級api |
IZKDateListener | PartitionsReassignedListener | KafkaController | ReassignPartitionsPath = "/admin/reassign_partitions" | |
IZKDateListener | LeaderChangeListener | ZookeeperLeaderElector | ControllerPath = "/controller" | Kafka的broker服務的升controller的競選目錄 |
IZKDateListener | ReassignedPartitionsIsrChangeListener | KafkaController | "/brokers/topics/${topic_name}/partitions/${partition_id}/state" | |
IZKDateListener | PartitionModificationsListener | PartitionStateMachine | "/brokers/topics/${topic_name} | |
IZkStateListener | SessionExpireListener | KafkaHealthcheck | kafka健康檢查會檢查進程和zk的session狀態(tài) | |
IZkStateListener | ZKSessionExpireListener | ZookeeperConsumerConnector | consumser初始化時注冊,在新建session時,也會調用到用syncedRebalance進行rebalance | |
IZkStateListener | SessionExpirationListener | KafkaController | kafkacontroller會監(jiān)聽與zk的鏈接狀態(tài),如果新建session,會調用onControllerResignation,并重新競選leader | |
IZkStateListener | ZkStateChangeListener | ZkNodeChangeNotificationListener In SimpleAclAuthorizer | AclChangedZkPath = "/kafka-acl-changes" AclChangedPrefix = "acl_changes_" | 在新建seesion時會通過注冊的notificaionhandler處理所有notifications |
IZkStateListener | ZkStateChangeListener | ZkNodeChangeNotificationListener In DynamicConfigManager | EntityConfigChangesPath = "/config/changes" EntityConfigChangeZnodePrefix = "config_change_" | 在新建seesion時會通過注冊的notificaionhandler處理所有notifications |
IZkStateListener | ZkSessionExpireListener | ZookeeperTopicEventWatcher | 消費時監(jiān)控和zk的鏈接狀態(tài),低級api有效 |
Kafka內部有大量的listener機制完成不同的功能。
listener接口 | listener | listener 模塊 | 監(jiān)控對象 | 備注 |
---|---|---|---|---|
IZkChildListener | DeleteTopicsListener | PartitionStateMachine | DeleteTopicsPath = "/admin/delete_topics" | |
IZkChildListener | IsrChangeNotificationListener | KafkaController | IsrChangeNotificationPath = "/isr_change_notification" | |
IZkChildListener | zkTopicEventListener | ZookeeperTOpicEventWatcher | BrokerTopicsPath = "/brokers/topics" | |
IZkChildListener | NodeChangeListener | ZkNodeCHangeNotificaionListener in SimpleAclAuthorizer | AclChangedZkPath = "/kafka-acl-changes" AclChangedPrefix = "acl_changes_" | |
IZkChildListener | NodeChangeListener | ZkNodeCHangeNotificaionListener In DynamicConfigManager | EntityConfigChangesPath = "/config/changes" EntityConfigChangeZnodePrefix = "config_change_" | |
IZkChildListener | ZKRebalancerListener | ZookeeperConsumerConnector | BrokerIdsPath = "/brokers/ids" | 同時也會被由ZKTopicPartitionChangeListener觸發(fā),然后調用syncedRebalance進行rebalance |
IZkChildListener | TopicChangeListener | PartitionStateMachine | BrokerTopicsPath = "/brokers/topics" | |
IZkChildListener | BrokerChangeListener | ReplicaStateMachine | BrokerIdsPath = "/brokers/ids" | |
IZKDateListener | PreferredReplicaElectionListener | KafkaController | PreferredReplicaLeaderElectionPath = "/admin/preferred_replica_election" | |
IZKDateListener | ZKTopicPartitionChangeListener | ZookeeperConsumerConnector | "/brokers/topics/${topic_name}" | consumer監(jiān)控的partition的變化并作出處理。 高級api |
IZKDateListener | PartitionsReassignedListener | KafkaController | ReassignPartitionsPath = "/admin/reassign_partitions" | |
IZKDateListener | LeaderChangeListener | ZookeeperLeaderElector | ControllerPath = "/controller" | Kafka的broker服務的升controller的競選目錄 |
IZKDateListener | ReassignedPartitionsIsrChangeListener | KafkaController | "/brokers/topics/${topic_name}/partitions/${partition_id}/state" | |
IZKDateListener | PartitionModificationsListener | PartitionStateMachine | "/brokers/topics/${topic_name} | |
IZkStateListener | SessionExpireListener | KafkaHealthcheck | kafka健康檢查會檢查進程和zk的session狀態(tài) | |
IZkStateListener | ZKSessionExpireListener | ZookeeperConsumerConnector | consumser初始化時注冊,在新建session時,也會調用到用syncedRebalance進行rebalance | |
IZkStateListener | SessionExpirationListener | KafkaController | kafkacontroller會監(jiān)聽與zk的鏈接狀態(tài),如果新建session,會調用onControllerResignation,并重新競選leader | |
IZkStateListener | ZkStateChangeListener | ZkNodeChangeNotificationListener In SimpleAclAuthorizer | AclChangedZkPath = "/kafka-acl-changes" AclChangedPrefix = "acl_changes_" | 在新建seesion時會通過注冊的notificaionhandler處理所有notifications |
IZkStateListener | ZkStateChangeListener | ZkNodeChangeNotificationListener In DynamicConfigManager | EntityConfigChangesPath = "/config/changes" EntityConfigChangeZnodePrefix = "config_change_" | 在新建seesion時會通過注冊的notificaionhandler處理所有notifications |
IZkStateListener | ZkSessionExpireListener | ZookeeperTopicEventWatcher | 消費時監(jiān)控和zk的鏈接狀態(tài),低級api有效 |