Flume1.6.0 ES2.3.5 sink

1.刪除flume1.6.0/lib里面的guava-.jar and jackson-core-.jar

2.拷貝es2.3.5/lib里es2.3.5.jar包,到flume1.6.0/lib

3.拷貝elasticsearch-sink2-1.0.jar,到flume1.6.0/lib

4.編輯flume1.6.0/conf/es.conf

5.啟動es

6.創建測試文件,并啟動flume

3.拷貝elasticsearch-sink2-1.0.jar,到flume1.6.0/lib

也可以build出來jar包
https://github.com/lucidfrontier45/ElasticsearchSink2

ElasticsearchSink2

This is Flume-NG Sink for Elasticsearch >= 2.0. I developed this because the official version does not support Elasticsearch >= 2.0 due to API changes. Hope Flume dev team will fix this soon.

Requirements

Flume-NG >= 1.6
Elasticsearch >= 2.0
Build

Build standard jar by the following command

$ ./gradlew build
Build fat jar which contains elasticsearch dependencies

$ ./gradlew assembly
Jar will be generated in build/libs

Usage

Append the built jar to Flume's classpath
remove guava-*.jar and jackson-core-*.jar in flume's default libs dir. They are outdated and newer version are included in Elasticsearch.
set sink type to com.frontier45.flume.sink.elasticsearch2.ElasticSearchSink in flume.conf
start flume agent

4.編輯flume1.6.0/conf/es.conf

agent.sources = tail
agent.channels = memoryChannel
agent.channels.memoryChannel.type = memory
agent.sources.tail.channels = memoryChannel
agent.sources.tail.type = exec
agent.sources.tail.command = tail -F /tmp/ttt

# Describe the sink ES
agent.sinks = elasticsearch
agent.sinks.elasticsearch.type = com.frontier45.flume.sink.elasticsearch2.ElasticSearchSink
agent.sinks.elasticsearch.hostNames = 127.0.0.1:9300
agent.sinks.elasticsearch.indexType = logs
agent.sinks.elasticsearch.indexName = flume
agent.sinks.elasticsearch.clusterName = elas
agent.sinks.elasticsearch.batchSize = 500
agent.sinks.elasticsearch.ttl = 5d
agent.sinks.elasticsearch.serializer = com.frontier45.flume.sink.elasticsearch2.ElasticSearchDynamicSerializer
agent.sinks.elasticsearch.indexNameBuilder = com.frontier45.flume.sink.elasticsearch2.TimeBasedIndexNameBuilder
agent.sinks.elasticsearch.channel = memoryChannel

5.啟動es

bin/elasticsearch

6.創建測試文件,并啟動flume

touch /tmp/ttt
echo "test" >> /tmp/ttt

bin/flume-ng agent -c . -f conf/es.conf -n agent -Dflume.root.logger=INFO,console

最后編輯于
?著作權歸作者所有,轉載或內容合作請聯系作者
平臺聲明:文章內容(如有圖片或視頻亦包括在內)由作者上傳并發布,文章內容僅代表作者本人觀點,簡書系信息發布平臺,僅提供信息存儲服務。

推薦閱讀更多精彩內容

  • 監控端口數據官方案例 需求 首先,Flume監控本機55566端口,然后通過telnet工具向本機55566端口發...
    ZFH__ZJ閱讀 505評論 0 2
  • Linux-Server-Notes PMS /home/softwareluke/圖片/2017-09-11 0...
    燕京博士閱讀 597評論 0 1
  • 本文所有內容皆關于 ES和 kibana 一、簡介 ELKB即elasticsearch、logstash、ki...
    閑云戲雨閱讀 1,571評論 0 0
  • 暑假的一天,媽媽帶我們出去玩。在一個清澈的池塘邊,打水漂的歡聲笑語,成了我難忘的記憶。 我們一共有...
    若淇丫頭閱讀 933評論 0 0
  • 和兩面人交往、聊天,受傷的人總是你,ta可能背后放個冷槍給你,中個陰箭什么的也不在話下,所謂真正的閨密很少遇到,至...
    伏霸天閱讀 238評論 0 0