學習了差不多一個星期，終于把flume-kafka-spark streaming貫通了，直接上流程圖：
至于為什么要這樣，當然是方便咯
參考某博客
一、環境部署
hadoop集群2.7.1
zookeerper集群
kafka集群：kafka_2.11-0.10.0.0
spark集群：spark-2.0.1-bin-hadoop2.7.tgz
flume1.7.0
環境搭建可參考我前面幾篇文章。不再贅述
三臺機器：master，slave1,slave2
二、配置flume
sink為kafka
source你隨意
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.port = 5140
a1.sources.r1.host = 192.168.31.131
a1.sources.r1.channels = c1
# Describe the sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = test5
a1.sinks.k1.brokerList = 192.168.31.131:9092
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.batchSize = 20
a1.sinks.k1.channel = c1
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
#yi bai tiao jiu submit
三、編程,KafkaWordCount.py
編寫spark steaming 代碼，讀取kafka流數據，并統計詞頻
# -*- coding: UTF-8 -*-
###spark streaming&&kafka
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
sc=SparkContext("local[2]","KafkaWordCount")
#處理時間間隔為1s
ssc=StreamingContext(sc,2)
zookeeper="192.168.31.131:2181,192.168.31.132:2181,192.168.31.133:2181"
#打開一個TCP socket 地址 和 端口號
topic={"test5":0,"test5":1,"test5":2}
groupid="test-consumer-group"
lines = KafkaUtils.createStream(ssc, zookeeper,groupid,topic)
lines1=lines.map(lambda x:x[1])
#對1s內收到的字符串進行分割
words=lines1.flatMap(lambda line:line.split(" "))
#映射為（word，1）元祖
pairs=words.map(lambda word:(word,1))
wordcounts=pairs.reduceByKey(lambda x,y:x+y)
#輸出文件，前綴+自動加日期
wordcounts.saveAsTextFiles("/tmp/fkafka")
wordcounts.pprint()
#啟動spark streaming應用
ssc.start()
#等待計算終止
ssc.awaitTermination()
四、啟動環境
1.啟動hadoop集群
start-all.sh
2.啟動spark集群
start-master.sh
start-slaves.sh
3.啟動zookeeper集群
在三臺機器下均輸入以下命令
zkServer.sh start
4.啟動kafka集群
在三臺機器下均輸入以下命令
kafka-server-start.sh -daemon ../config/server.properties
5.jps查看進程
master:
slave1與slave2一樣:
確保所有進程啟動
6.創建kafka topic
kafka-topics.sh --create --zookeeper 192.168.31.131:2181,192.168.31.132:2181,192.168.31.133:2181 --replication-factor 3 --partitions 3 --topic test5
7.啟動flume agent
flume-ng agent --conf flume/conf/ -f /home/cms/flume/conf/flume-conf.properties -n a1 -Dflume.root.logger=INFO,console
五、測試
1.運行KafkaWordCount.py
在master下
運行
spark-submit --jars kafka/libs/spark-streaming-kafka-0-8-assembly_2.11-2.0.1.jar KafkaWordCount.py 2> error.txt
2.發送數據
echo "hello'\t'word" | nc 192.168.31.131 5140
3.觀察終端輸出
六、hdfs上查看輸出
hadoop fs -ls /tmp/fkafka*
參考文檔
(flume-kafka- spark streaming(pyspark) - redis 實時日志收集實時計算)[http://blog.csdn.net/zhong_han_jun/article/details/50721736]
http://spark.apache.org/docs/latest/streaming-kafka-0-8-integration.html
