1）ELK5.1+Redis+Filebeat日志系統的部署與測試

一，文件說明

  • centos7
  • jdk-8u112-linux-x64.tar.gz
  • elasticsearch-5.1.1.tar.gz
  • filebeat-5.1.1-x86_64.rpm
  • kibana-5.1.1-linux-x86_64.tar.gz
  • logstash-5.1.1.tar.gz
  • redis-3.2.6.tar.gz

二，環境配置

elk5.1之后需要jdk1.8以上的支持，所以需要配置jdk環境變量

1 用java -version命令查看centos是否自帶openJDK 如果自帶則卸載，卸載命令如下：
rpm -e --nodeps 'rpm -qa | grep java'
2 解壓,重命名jdk到 /usr/local/java目錄 并記錄了jdk根目錄

# tar -zxf jdk-8u112-linux-x64.tar.gz
# mv jdk1.8.0_112/ jdk1.8/
# pwd
/usr/local/java/jdk1.8

3 配置java環境變量

# vi /etc/profile
加入如下內容:
export NODE_HOME=/opt/soft/node
export PATH=$NODE_HOME/bin:$PATH # 這是node.js的環境變量

export JAVA_HOME=/usr/local/java/jdk1.8
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin

# source /etc/profile
# java -version
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

三 解壓并授權elk文件

# tar -zxf elasticsearch-5.1.1.tar.gz
# tar -zxf kibana-5.1.1-linux-x86_64.tar.gz
# tar -zxf logstash-5.1.1.tar.gz

# chmod -R 777 elasticsearch-5.1.1
# chmod -R 777 kibana-5.1.1-linux-x86_64
# chmod -R 777 logstash-5.1.1

四 新建centos系統用戶并授予root權限,防火墻關閉

# useradd elkuser
# passwd elkuser

授予root權限

# vi /etc/sudoers 在root   ALL=(ALL)       ALL后添加
elkuser    ALL=(ALL)       ALL
注意:強行寫入用 wq!
# source /etc/sudoers 使文件生效

關閉防火墻

查看狀態
# firewall-cmd --state
# service iptables status
centos7 如果沒有安裝iptables-services  那么第二條命令不生效
如果是運行狀態則關閉(關閉方式請自行百度)

五 安裝配置filebeat

進入filebeat文件目錄,如果不成功,則去掉sudo

sudo rpm -vi filebeat-5.1.2-x86_64.rpm

查看filebeat安裝位置 rpm -ql filebeat

配置filebeat

###################### Filebeat Configuration Example #########################
#=========================== Filebeat prospectors =============================

filebeat.prospectors:

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/elklog/Linux/*.log
  document_type: linux
- input_type: log  
  paths:
    - /var/elklog/openstackMitakaLog/*.log
  document_type: api

#================================ Outputs =============================

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.1.19:5043"]

啟動filbeat

sudo /etc/init.d/filebeat start
或 /etc/init.d/filebeat start

結果:Starting filebeat: 2017/01/16 11:19:40.838639 beat.go:267: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2017/01/16 11:19:40.838687 beat.go:177: INFO Setup Beat: filebeat; Version: 5.1.1
2017/01/16 11:19:40.838827 logp.go:219: INFO Metrics logging every 30s
2017/01/16 11:19:40.838824 logstash.go:90: INFO Max Retries set to: 3
2017/01/16 11:19:40.839014 outputs.go:106: INFO Activated logstash as output plugin.
2017/01/16 11:19:40.839337 publish.go:291: INFO Publisher name: localhost.localdomain
2017/01/16 11:19:40.847865 async.go:63: INFO Flush Interval set to: 1s
2017/01/16 11:19:40.847879 async.go:64: INFO Max Bulk Size set to: 2048
Config OK                                                         [確定]

六 配置運行elasticsearch

配置es(都以此為簡稱) 沒配置的即為默認配置

# ======================== Elasticsearch Configuration =========================
# ---------------------------------- Cluster -----------------------------------
cluster.name: bjhit-cluster
# ------------------------------------ Node ------------------------------------
node.name: node-bjhit
# ---------------------------------- Network -----------------------------------
network.host: 192.168.1.19
http.port: 9200

運行es： 進入elasticsearch解壓文件目錄
es不能用root用戶啟動,只能用我們剛剛建立的elkuser來啟動否則或報錯

./bin/elasticsearch

七 安裝運行redis

安裝redis需要依賴gcc 所以先確定gcc安裝了 如果沒有則請安裝;確認安裝命令如下:

確認：gcc -v
安裝：yum install -y gcc g++ gcc-c++ make
完成gcc安裝后:

$ tar xzf redis-3.2.6.tar.gz
$ cd redis-3.2.6
$ make
啟動測試redis
$ src/redis-server
另開窗口測試
$ src/redis-cli
redis> set foo bar
OK
redis> get foo
"bar"

測試成功后,啟動redis 如果redis安裝在另一個linux下的話則運行:

./src/redis-server --protected-mode no 否則直接運行./src/redis-server

八 安裝logstash插件以及配置啟動logstash

filebeat和logstash之間必須要在logstash目錄下安裝一個插件:

進入logstash目錄后執行： ./bin/logstash-plugin install logstash-input-beats
安裝過程可能不成功，那就堅持不懈再次安裝，最后會成功的；以下是成功報文：

Validating logstash-input-beats
Installing logstash-input-beats
WARNING: SSLSocket#session= is not supported
Installation successful

配置logstash 新建且配置完后放到logstash的bin目錄:

1,redis-input.conf

#input { stdin { } }
#input {
#    file {
#        path => ["/var/elklog/*.log"]
#        type => "system"
#        start_position => "beginning"
#    }
#}
input {
 beats {
    port => "5043"
 }
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
   #  elasticsearch {
   #  hosts => [ "192.168.1.19:9200" ]
#   index => "logstash-%{type}-%{+YYYY.MM.dd}"
 #       document_type => "%{type}"
  #      flush_size => 20000
   #     idle_flush_time => 10
    #    template_overwrite => true
   # }

   redis {
        host => "192.168.1.211"
        port => 6379
        data_type => "list"
        key => "logstash:redis"
    }
    stdout { codec => rubydebug }

   
}

2,redis-output.conf

#input { stdin { } }
#input {
#    file {
#        path => ["/var/elklog/*.log"]
#        type => "system"
#        start_position => "beginning"
#    }
#}
#input {
# beats {
#    port => "5043"
# }
#}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
input {
    redis {
        data_type => "list"
        key => "logstash:redis"
        host => "192.168.1.211"
        port => 6379
    }
}

output {
    if[type] =="linux"{
     elasticsearch {
        hosts => [ "192.168.1.19:9200" ]
    index => "linux-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        flush_size => 20000
        idle_flush_time => 10
        template_overwrite => true
    }
   }else if[type] =="api"{
     elasticsearch {
        hosts => [ "192.168.1.19:9200" ]
        index => "api-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        flush_size => 20000
        idle_flush_time => 10
        template_overwrite => true
    }
   }


    stdout { codec => rubydebug }

   
}

啟動logstash
分別用如下參數啟動
./logstash -f redis-output.conf
./logstash -f redis-input.conf

九配置啟動kibana

配置kibana

server.port: 5601
server.host: "192.168.1.19"
server.name: "localhost"
elasticsearch.url: "http://192.168.1.19:9200"

啟動kibana
進入kibana目錄(我測試的時候elk都是用的elkuser用戶來運行)
./bin/kibana

最后編輯于 ：
?著作權歸作者所有,轉載或內容合作請聯系作者
平臺聲明：文章內容（如有圖片或視頻亦包括在內）由作者上傳并發布，文章內容僅代表作者本人觀點，簡書系信息發布平臺，僅提供信息存儲服務。

推薦閱讀更多精彩內容