Kakfa溝通機制(APiKeys & Listener)

Kafka不同進程“溝通機制”

Kafka服務的定位是一種高吞吐量的分布式消息訂閱系統(tǒng)。服務再運行過程中,不同的進程(broker和controller之間,客戶端和集群)之間需要進行“溝通”來保證功能的可用性。

Kafka主要通過兩種方式進行“溝通”,以保證自身狀態(tài)或需求被感知:

通過nio方式完成“溝通”

controller通過ControllerChannelManager發(fā)送消息到各broker,主要有ApiKeys.LEADER_AND_ISR, ApiKeys.STOP_REPLICA, ApiKeys.UPDATE_METADATA_KEY 三種消息。其中僅有ApiKeys.STOP_REPLICA有相關callback操作。broker通過KafkaRequestHandlerPool接收消息并調用kafkaApi進行消息相應。其他如producer發(fā)送消息,consumser消費消息,獲取集群信息等操作,也是客戶端通過nio的方式發(fā)送請求。Broker可處理的消息有如下:

消息 備注
ApiKeys.LEADER_AND_ISR Controller 發(fā)送給broker
ApiKeys.STOP_REPLICA Controller 發(fā)送給broker
ApiKeys.UPDATE_METADATA_KEY Controller 發(fā)送給broker
ApiKeys.PRODUCE producer發(fā)送生產(chǎn)數(shù)據(jù)請求給broker
ApiKeys.FETCH consumer發(fā)送消費數(shù)據(jù)請求給broker
ApiKeys.LIST_OFFSETS 客戶端/consumer 發(fā)送查詢offset請求
ApiKeys.METADATA 獲取集群topic信息
ApiKeys.CONTROLLED_SHUTDOWN_KEY brokershutdown時向controller發(fā)送消息
ApiKeys.OFFSET_COMMIT consumer向coordinator發(fā)送的offset commit 請求
ApiKeys.OFFSET_FETCH consumer發(fā)送的請求partitions的當前的offset
ApiKeys.GROUP_COORDINATOR consumer向broker發(fā)送group的coordinator是哪個節(jié)點的請求
ApiKeys.JOIN_GROUP consumer向groupcoordinator發(fā)送 join group的請求
ApiKeys.HEARTBEAT consumer定期發(fā)送heartbeat請求到groupcoordinator
ApiKeys.LEAVE_GROUP consumer停止消費時發(fā)送退出group請求到groupcoordinator
ApiKeys.SYNC_GROUP consumer變?yōu)閘eader獲取follower時發(fā)送的已同步請求到groupcoordinator
ApiKeys.DESCRIBE_GROUPS 客戶端工具kafka-consumer-groups.sh ConsumerGroupCommand中使用describe group信息所用
ApiKeys.LIST_GROUPS 客戶端工具kafka-consumer-groups.sh ConsumerGroupCommand中使用list group信息所用
ApiKeys.SASL_HANDSHAKE nio的select發(fā)送消息/請求時,需要先發(fā)送sasl——handshake請求
ApiKeys.API_VERSIONS 客戶端發(fā)送的kafka版本請求

通過zookeeper注冊監(jiān)聽器完成“溝通”

客戶端(broker/client)如果需要將訴求(如topic創(chuàng)建,刪除,reblance,partition reassign等)告知服務,所采取的方案是再ZooKeeper中創(chuàng)建目錄,寫入數(shù)據(jù),服務端(此處指controller)通過listener監(jiān)控到zk中的變化,并作出響應

Zookeeper的watch機制與第三方封裝

  • Zookeeper是分布式應用程序協(xié)調服務。不同的進程通過約定的目錄的寫入和監(jiān)控操作完成消息的互相感知。開源大數(shù)據(jù)多數(shù)服務的高可用性實現(xiàn)都是依靠zookeeper的協(xié)調功能完成。zookeeper客戶端通過watcher機制完成對zookeeper的集群的監(jiān)控。

  • 由于zookeeper應用開發(fā)的復雜性,出現(xiàn)了第三方對zk的應用開發(fā)的封裝。常用的有Curator FrameWork,I0Itec-zkclient。其中hive,hbase,spark等都是通過Curator FrameWork完成和zk的交互。Kafka則選用的是I0Itec-zkclient與zk交互。

Kafka中實現(xiàn)listener的方法:

  • Kafka通過zkUtils中調用org.I0Itec.zkclient.zkCLient完成listener(listener中相關的handle方法有相關的處理邏輯)的注冊。
  • zkClient繼承Watcher,再監(jiān)控的對象發(fā)生變化時,zkCLient中的process方法會被調用
  • 在該方法中會完成對不同的listener的handle方法的調用

I0Itec-zkclient主要有如下三種listener,分別實現(xiàn)不同的監(jiān)聽方式。

消息 handle的方法 備注
IZkChildListener handleChildChange listening on zk child changes for a given path
IZkDataListener handleDataChange listening on zk data changes for a given path
IZkDataListener handleDataDeleted listening on zk data changes for a given path
IZkStateListener handleStateChanged Called when the zookeeper connection state has changed.
IZkStateListener handleNewSession Called after the zookeeper session has expired and a new session has been created
IZkStateListener handleSessionEstablishmentError Called when a session cannot be re-established
listener接口 listener listener 模塊 監(jiān)控對象 備注
IZkChildListener DeleteTopicsListener PartitionStateMachine DeleteTopicsPath = "/admin/delete_topics"
IZkChildListener IsrChangeNotificationListener KafkaController IsrChangeNotificationPath = "/isr_change_notification"
IZkChildListener zkTopicEventListener ZookeeperTOpicEventWatcher BrokerTopicsPath = "/brokers/topics"
IZkChildListener NodeChangeListener ZkNodeCHangeNotificaionListener in SimpleAclAuthorizer AclChangedZkPath = "/kafka-acl-changes" AclChangedPrefix = "acl_changes_"
IZkChildListener NodeChangeListener ZkNodeCHangeNotificaionListener In DynamicConfigManager EntityConfigChangesPath = "/config/changes" EntityConfigChangeZnodePrefix = "config_change_"
IZkChildListener ZKRebalancerListener ZookeeperConsumerConnector BrokerIdsPath = "/brokers/ids" 同時也會被由ZKTopicPartitionChangeListener觸發(fā),然后調用syncedRebalance進行rebalance
IZkChildListener TopicChangeListener PartitionStateMachine BrokerTopicsPath = "/brokers/topics"
IZkChildListener BrokerChangeListener ReplicaStateMachine BrokerIdsPath = "/brokers/ids"
IZKDateListener PreferredReplicaElectionListener KafkaController PreferredReplicaLeaderElectionPath = "/admin/preferred_replica_election"
IZKDateListener ZKTopicPartitionChangeListener ZookeeperConsumerConnector "/brokers/topics/${topic_name}" consumer監(jiān)控的partition的變化并作出處理。 高級api
IZKDateListener PartitionsReassignedListener KafkaController ReassignPartitionsPath = "/admin/reassign_partitions"
IZKDateListener LeaderChangeListener ZookeeperLeaderElector ControllerPath = "/controller" Kafka的broker服務的升controller的競選目錄
IZKDateListener ReassignedPartitionsIsrChangeListener KafkaController "/brokers/topics/${topic_name}/partitions/${partition_id}/state"
IZKDateListener PartitionModificationsListener PartitionStateMachine "/brokers/topics/${topic_name}
IZkStateListener SessionExpireListener KafkaHealthcheck kafka健康檢查會檢查進程和zk的session狀態(tài)
IZkStateListener ZKSessionExpireListener ZookeeperConsumerConnector consumser初始化時注冊,在新建session時,也會調用到用syncedRebalance進行rebalance
IZkStateListener SessionExpirationListener KafkaController kafkacontroller會監(jiān)聽與zk的鏈接狀態(tài),如果新建session,會調用onControllerResignation,并重新競選leader
IZkStateListener ZkStateChangeListener ZkNodeChangeNotificationListener In SimpleAclAuthorizer AclChangedZkPath = "/kafka-acl-changes" AclChangedPrefix = "acl_changes_" 在新建seesion時會通過注冊的notificaionhandler處理所有notifications
IZkStateListener ZkStateChangeListener ZkNodeChangeNotificationListener In DynamicConfigManager EntityConfigChangesPath = "/config/changes" EntityConfigChangeZnodePrefix = "config_change_" 在新建seesion時會通過注冊的notificaionhandler處理所有notifications
IZkStateListener ZkSessionExpireListener ZookeeperTopicEventWatcher 消費時監(jiān)控和zk的鏈接狀態(tài),低級api有效

Kafka內部有大量的listener機制完成不同的功能。

listener接口 listener listener 模塊 監(jiān)控對象 備注
IZkChildListener DeleteTopicsListener PartitionStateMachine DeleteTopicsPath = "/admin/delete_topics"
IZkChildListener IsrChangeNotificationListener KafkaController IsrChangeNotificationPath = "/isr_change_notification"
IZkChildListener zkTopicEventListener ZookeeperTOpicEventWatcher BrokerTopicsPath = "/brokers/topics"
IZkChildListener NodeChangeListener ZkNodeCHangeNotificaionListener in SimpleAclAuthorizer AclChangedZkPath = "/kafka-acl-changes" AclChangedPrefix = "acl_changes_"
IZkChildListener NodeChangeListener ZkNodeCHangeNotificaionListener In DynamicConfigManager EntityConfigChangesPath = "/config/changes" EntityConfigChangeZnodePrefix = "config_change_"
IZkChildListener ZKRebalancerListener ZookeeperConsumerConnector BrokerIdsPath = "/brokers/ids" 同時也會被由ZKTopicPartitionChangeListener觸發(fā),然后調用syncedRebalance進行rebalance
IZkChildListener TopicChangeListener PartitionStateMachine BrokerTopicsPath = "/brokers/topics"
IZkChildListener BrokerChangeListener ReplicaStateMachine BrokerIdsPath = "/brokers/ids"
IZKDateListener PreferredReplicaElectionListener KafkaController PreferredReplicaLeaderElectionPath = "/admin/preferred_replica_election"
IZKDateListener ZKTopicPartitionChangeListener ZookeeperConsumerConnector "/brokers/topics/${topic_name}" consumer監(jiān)控的partition的變化并作出處理。 高級api
IZKDateListener PartitionsReassignedListener KafkaController ReassignPartitionsPath = "/admin/reassign_partitions"
IZKDateListener LeaderChangeListener ZookeeperLeaderElector ControllerPath = "/controller" Kafka的broker服務的升controller的競選目錄
IZKDateListener ReassignedPartitionsIsrChangeListener KafkaController "/brokers/topics/${topic_name}/partitions/${partition_id}/state"
IZKDateListener PartitionModificationsListener PartitionStateMachine "/brokers/topics/${topic_name}
IZkStateListener SessionExpireListener KafkaHealthcheck kafka健康檢查會檢查進程和zk的session狀態(tài)
IZkStateListener ZKSessionExpireListener ZookeeperConsumerConnector consumser初始化時注冊,在新建session時,也會調用到用syncedRebalance進行rebalance
IZkStateListener SessionExpirationListener KafkaController kafkacontroller會監(jiān)聽與zk的鏈接狀態(tài),如果新建session,會調用onControllerResignation,并重新競選leader
IZkStateListener ZkStateChangeListener ZkNodeChangeNotificationListener In SimpleAclAuthorizer AclChangedZkPath = "/kafka-acl-changes" AclChangedPrefix = "acl_changes_" 在新建seesion時會通過注冊的notificaionhandler處理所有notifications
IZkStateListener ZkStateChangeListener ZkNodeChangeNotificationListener In DynamicConfigManager EntityConfigChangesPath = "/config/changes" EntityConfigChangeZnodePrefix = "config_change_" 在新建seesion時會通過注冊的notificaionhandler處理所有notifications
IZkStateListener ZkSessionExpireListener ZookeeperTopicEventWatcher 消費時監(jiān)控和zk的鏈接狀態(tài),低級api有效
最后編輯于
?著作權歸作者所有,轉載或內容合作請聯(lián)系作者
平臺聲明:文章內容(如有圖片或視頻亦包括在內)由作者上傳并發(fā)布,文章內容僅代表作者本人觀點,簡書系信息發(fā)布平臺,僅提供信息存儲服務。

推薦閱讀更多精彩內容

  • Spring Cloud為開發(fā)人員提供了快速構建分布式系統(tǒng)中一些常見模式的工具(例如配置管理,服務發(fā)現(xiàn),斷路器,智...
    卡卡羅2017閱讀 134,881評論 18 139
  • 姓名:周小蓬 16019110037 轉載自:http://blog.csdn.net/YChenFeng/art...
    aeytifiw閱讀 34,743評論 13 425
  • ** 今天看了一下kafka官網(wǎng),嘗試著在自己電腦上安裝和配置,然后學一下官方document。** Introd...
    RainChang閱讀 5,039評論 1 30
  • 本文轉載自http://dataunion.org/?p=9307 背景介紹Kafka簡介Kafka是一種分布式的...
    Bottle丶Fish閱讀 5,491評論 0 34
  • kafka的定義:是一個分布式消息系統(tǒng),由LinkedIn使用Scala編寫,用作LinkedIn的活動流(Act...
    時待吾閱讀 5,356評論 1 15