linux搭建,kafkao3節點虛擬機為CentOS6,ip為192.168.1.128，192.168.1.129和192.168.1.130，域名分別為master,worker1,worker2
1. 集群
#192.168.1.128[root@masterlocal]#cd/home/gilbert/app/rar/[root@masterrar]#tarzxvfkafka_2.10-0.10.2.0.tgz[root@masterrar]#mvkafka_2.10-0.10.2.0/home/gilbert/app/kafka?
配置文件路徑：kafka/config/server.properties
修改配置文件
啟動kafka? ?
[root@masterkafka]#./bin/kafka-server-start.shconfig/server.properties&[2018-06-2502:31:21,931]INFOKafkaConfigvalues:advertised.host.name=nulladvertised.listeners=nulladvertised.port=nullauthorizer.class.name=auto.create.topics.enable=trueauto.leader.rebalance.enable=truebackground.threads=10broker.id=0broker.id.generation.enable=truebroker.rack=nullcompression.type=producerconnections.max.idle.ms=600000controlled.shutdown.enable=truecontrolled.shutdown.max.retries=3controlled.shutdown.retry.backoff.ms=5000controller.socket.timeout.ms=30000
創建topic
#創建topictopic名字為gilbert[root@masterkafka]#./bin/kafka-topics.sh--create--zookeepermaster:2181,worker1:2181,worker2:2181--replication-factor3--partitions3--topicgilbertCreatedtopic"gilbert".
查看topic
[root@masterkafka]#./bin/kafka-topics.sh--describe--zookeepermaster:2181,worker1:2181,worker2:2181--topicgilbertTopic:gilbertPartitionCount:3ReplicationFactor:3Configs:Topic:gilbertPartition:0Leader:2Replicas:2,0,1Isr:2,0,1Topic:gilbertPartition:1Leader:0Replicas:0,1,2Isr:0,1,2Topic:gilbertPartition:2Leader:1Replicas:1,2,0Isr:1,2,0 ? ? ? ? ?[root@masterkafka]#./bin/kafka-topics.sh--list--zookeepermaster:2181,worker1:2181,worker2:2181gilberttest
創建producer
./bin/kafka-console-producer.sh--broker-listmaster:9092-topicgilbert
創建consumer，分別在3臺服務器上執行創建消費者
#192.168.1.128服務器[root@masterkafka]#./bin/kafka-console-consumer.sh--zookeepermaster:2181,worker1:2181,worker2:2181-topicgilbert--from-beginningUsingtheConsoleConsumerwitholdconsumerisdeprecatedandwillberemovedinafuturemajorrelease.Considerusingthenewconsumerbypassing[bootstrap-server]insteadof[zookeeper]. ?#192.168.1.129服務器[root@worker1kafka_2.10-0.10.2.0]#./bin/kafka-console-consumer.sh--zookeepermaster:2181,worker1:2181,worker2:2181-topicgilbert--from-beginningUsingtheConsoleConsumerwitholdconsumerisdeprecatedandwillberemovedinafuturemajorrelease.Considerusingthenewconsumerbypassing[bootstrap-server]insteadof[zookeeper].?#192.168.1.130服務器[root@worker2kafka_2.10-0.10.2.0]#./bin/kafka-console-consumer.sh--zookeepermaster:2181,worker1:2181,worker2:2181-topicgilbert--from-beginningUsingtheConsoleConsumerwitholdconsumerisdeprecatedandwillberemovedinafuturemajorrelease.Considerusingthenewconsumerbypassing[bootstrap-server]insteadof[zookeeper].??
在#192.168.1.128服務器上生產者控制臺輸入：hello kafka進行測試
在3臺服務器上的消費者都正常接收到消息
刪除topic
[root@masterkafka]#./bin/kafka-topics.sh--delete--zookeepermaster:2181,worker1:2181,worker2:2181--topictestTopictestismarkedfordeletion.Note:Thiswillhavenoimpactifdelete.topic.enableisnotsettotrue
springboot集成kafka
1.生產者kafka-producer
a) pom文件
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter</artifactId></dependency><dependency><groupId>org.springframework.kafka</groupId><artifactId>spring-kafka</artifactId></dependency><dependency><groupId>com.google.code.gson</groupId><artifactId>gson</artifactId><version>2.8.2</version></dependency><dependency><groupId>org.projectlombok</groupId><artifactId>lombok</artifactId><optional>true</optional></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-test</artifactId><scope>test</scope></dependency>
b) yml配置文件,本例為kafka3節點集群
spring:? kafka: ?? bootstrap-servers: http://master:9092,http://worker1:9092,http://worker2:9092 ?? producer: ? ?? retries: 0 ? ?? batch-size: 16384 ? ?? buffer-memory: 33554432 ? ?? key-serializer: org.apache.kafka.common.serialization.StringSerializer ? ?? value-serializer: org.apache.kafka.common.serialization.StringSerializer
c) message消息實體類
@DatapublicclassMessage{privateLongid;//idprivateStringmsg;//消息privateDatesendTime;//時間戳}
d) 生產者
@Component@Slf4jpublicclassKafkaProducer{?@AutowiredprivateKafkaTemplate<String,String>kafkaTemplate;?privateGsongson=newGsonBuilder().create();?//發送消息方法publicvoidsend() {Messagemessage=newMessage();message.setId(System.currentTimeMillis());message.setMsg(UUID.randomUUID().toString());message.setSendTime(newDate());log.info("+++++++++++++++++++++? message = {}",gson.toJson(message));? //topic-ideal為主題kafkaTemplate.send("topic-ideal",gson.toJson(message)); ?? }}
e) 測試類,運行kafkaProducer方法即可
@RunWith(SpringRunner.class)@SpringBootTestpublicclassKafkaProducerApplicationTests{?? ? @Autowired? ? privateKafkaProducerkafkaProducer;?? ? @Test? ? publicvoidkafkaProducer(){? ? ? ? this.kafkaProducer.send();? ? }?? ? @Test? ? publicvoidcontextLoads() {? ? }?}
2. 消費者kafka-consumer
a) pom文件
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter</artifactId></dependency><dependency><groupId>org.springframework.kafka</groupId><artifactId>spring-kafka</artifactId></dependency><dependency><groupId>com.google.code.gson</groupId><artifactId>gson</artifactId><version>2.8.2</version></dependency><dependency><groupId>org.projectlombok</groupId><artifactId>lombok</artifactId><optional>true</optional></dependency>
b) yml配置文件
server:? port: 9999spring:? kafka: ?? bootstrap-servers: http://master:9092,http://worker1:9092,http://worker2:9092 ?? consumer: ? ?? group-id: ideal-consumer-group ? ?? auto-offset-reset: earliest ? ?? enable-auto-commit: true ? ?? auto-commit-interval: 20000 ? ?? key-deserializer: org.apache.kafka.common.serialization.StringDeserializer ? ?? value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
c) 消費者類
@Component@Slf4jpublicclassKafkaConsumer{?@KafkaListener(topics={"topic-ideal"})publicvoidconsumer(ConsumerRecord<?,?>record){Optional<?>kafkaMessage=Optional.ofNullable(record.value());if(kafkaMessage.isPresent()) {Objectmessage=kafkaMessage.get();log.info("----------------- record ="+record);log.info("------------------ message ="+message); ? ? ?? } ?? }}
運行消費者kafka-consumer，再運行kafka-producer工程測試類KafkaProducerApplicationTests中kafkaProducer()方法,可以看到消費者后臺正常接收消息
