Elasticsearch, Logstash 和 Kibana 簡稱ELK，是日志平臺。
用途：Bug定位，歷史統計。
自從有了Dokcer 什么都想往里面放來著。

1. 規劃

ELK.jpg

2. 基礎鏡像

因ELK的啟動需要用到jdk，不想每個docker都重復配置jdk，提前準備elk的基礎鏡像

• 準備工作 下載jdk1.8
• 編寫dockerfile

FROM centos:latest
MAINTAINER chances-lixr
VOLUME [ "/opt/product/data/" ]
RUN  /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN /bin/echo -e "ZONE="Asia/Shanghai"\nUTC=false\nRTC=false" > /etc/sysconfig/clock
RUN mkdir /opt/product/tools/
ADD ./tools /opt/product/tools/
ENV JAVA_HOME /opt/product/tools/jdk1.8.0_51
CMD ["/usr/sbin/init"]

• docker build -t elkbase:v1.0 ./
  得到一個ELK的基礎鏡像。

3. Elasticsearch 配置

ELK的官方網站 https://www.elastic.co

• 下載Elasticsearch tar包
• 編寫dockerfile

FROM elkbase:v1.0
MAINTAINER chances-lixr
VOLUME [ "/opt/product/data/" ]
ADD ./elk /opt/product/
RUN useradd elk && chown -R elk:elk /opt/product/elasticsearch-5.1.2
ADD all.sh /root/
RUN chmod +x /root/all.sh
EXPOSE 9200
EXPOSE 9300
ENTRYPOINT ["/root/all.sh"]
CMD ["/usr/sbin/init"]

• 編寫啟動運行腳本

#!/bin/bash
echo "* soft nofile 65536" > /etc/security/limits.conf 
echo "* hard nofile 131072" > /etc/security/limits.conf
echo "* soft nproc 2048" > /etc/security/limits.conf
echo "* hard nproc 4096" > /etc/security/limits.conf
echo "vm.max_map_count=655360" > /etc/sysctl.conf 
sysctl -p
cd /opt/product/elasticsearch-5.1.2/config/
rm -rf elasticsearch.yml
cp /opt/product/data/elk/elasticsearch.yml .
chown -R elk:elk /opt/product/elasticsearch-5.1.2
chown -R elk:elk /opt/product/data/elk/elasticsearchdata
su - elk <<!
export JAVA_HOME=/opt/product/tools/jdk1.8.0_51
export PATH=$JAVA_HOME/bin:$PATH
/opt/product/elasticsearch-5.1.2/bin/elasticsearch 

• docker build -f elasticsearch:v1.0 ./

• elasticsearch的配置
  在/opt/product/data目錄下
  建立elk目錄，并拷貝elasticsearch.yml文件到該目錄
  設置
  path.data: /opt/product/data/elk/elasticsearchdata
  network.host: 0.0.0.0
  在/opt/product/data目錄創建elasticsearchdata 目錄

• 啟動

docker run --privileged --restart=always -d -ti -v /opt/product/data:/opt/product/data -p 9200:9200 -p 9300:9300 elasticsearch:v1.0 /bin/bash

• 訪問
  http://ip:9200/kimchy/tweet/1?pretty%27%20-d
  看到有json串返回即可

4. Logstash 配置

• 下載Logstash tar包
• 編寫dockerfile

FROM elkbase:v1.0
MAINTAINER chances-lixr
VOLUME [ "/opt/product/data/" ]
ADD ./elk /opt/product/
ADD all.sh /root/
RUN chmod +x /root/all.sh
EXPOSE 5044
EXPOSE 4560
EXPOSE 8080
ENTRYPOINT ["/root/all.sh"]
CMD ["/usr/sbin/init"]

• 編寫啟動運行腳本

#!/bin/bash
export JAVA_HOME=/opt/product/tools/jdk1.8.0_51
export PATH=$JAVA_HOME/bin:$PATH
JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=UTF8  -Duser.timezone=GMT+08"
cd /opt/product/logstash-5.1.2/config/
rm -rf logstash.yml
cp /opt/product/data/elk/logstash.yml logstash.yml
/opt/product/logstash-5.1.2/bin/logstash -f /opt/product/data/elk/logstash.conf

• docker build -f logstash:v1.0 ./
• 配置logstash
  在/opt/product/data/elk目錄下創建logstash.conf

input {     
   beats {
    port => "5044"
   }
}
output {
   elasticsearch {
       hosts => ["elasticsearch的ip:9200"]
       index => "logstash-tomcat-accesslog-%{+YYYY.MM.dd}"
    }       
}

logstash.yml

將logstash本身的logstash.yml 拷貝到/opt/product/data/elk目錄下
在/opt/product/data/elk目錄下建立 logstashdata目錄

• 啟動logstash

docker run  --restart=always -d -ti -v /opt/product/data:/opt/product/data -p 5044:5044 -p 4560:4560 -p 18080:8080 logstash:v1.0 /bin/bash

5. kibana 配置

• 下載kibana
• 編寫dockerfile

FROM elkbase:v1.0
MAINTAINER chances-lixr
VOLUME [ "/opt/product/data/" ]
ADD ./elk /opt/product/
ADD all.sh /root/
RUN chmod +x /root/all.sh
EXPOSE 5601
ENTRYPOINT ["/root/all.sh"]
CMD ["/usr/sbin/init"]

• 編寫運行文件

#!/bin/bash 
export JAVA_HOME=/opt/product/tools/jdk1.8.0_51
export PATH=$JAVA_HOME/bin:$PATH
cd /opt/product/kibana-5.1.2/config/
rm -rf kibana.yml
ln -s /opt/product/data/elk/kibana.yml . 
cd /opt/product/kibana-5.1.2/
rm -rf data
ln -s  /opt/product/data/elk/kibanadata /opt/product/kibana-5.1.2/data
/opt/product/kibana-5.1.2/bin/kibana 

• docker build -t kibana:v1.0 ./
• 配置
  在/opt/product/data/elk目錄下建立kibana.yml

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://ip:9200"

• 啟動

docker run  --restart=always -d -ti -v /opt/product/data:/opt/product/data -p 5601:5601  kibana:v1.0 /bin/bash

• 訪問
  http://ip:5601/

6. FileBeat配置

• 下載Filebeat
• 編寫dockerfile

FROM elkbase:v1.0
MAINTAINER chances-lixr
VOLUME [ "/opt/product/data/" ]
ADD ./elk /opt/product/
ADD all.sh /root/
RUN chmod +x /root/all.sh
ENTRYPOINT ["/root/all.sh"]
CMD ["/usr/sbin/init"]

• 編寫運行腳本

#!/bin/bash
export JAVA_HOME=/opt/product/tools/jdk1.8.0_51
export PATH=$JAVA_HOME/bin:$PATH
cd /opt/product/filebeat-5.1.2/
rm -rf filebeat.yml
ln -s /opt/product/data/elk/filebeat.yml . 
rm -rf data 
ln -s /opt/product/data/elk/filebeatdata /opt/product/filebeat-5.1.2/data
/opt/product/filebeat-5.1.2/filebeat -e -c filebeat.yml

• docker build -t filebeat:v1.0 ./
• 配置filebeat
  在/opt/product/data/elk目錄下 創建
  filebeat.yml

filebeat.prospectors:
  - input_type: log
    document_type: tomcataccess
  paths:
     - /opt/product/data/logs/tomcat/localhost_access_log*.txt
     - /opt/product/data/epg2logs/tomcat/localhost_access_log*.txt
output.logstash:
  # The Logstash hosts
  hosts: ["ip:5044"]

在/opt/product/data/elk 目錄下創建filebeatdata

• 啟動filebeat
  filebeat.yml

docker run -d -ti -v /opt/product/data:/opt/product/data  filebeat:v1.0 /bin/bash

7. UI界面

• 訪問http://ip:5601/
• 配置Configure an index pattern
  因當前是filebeat收集日志，Index name or pattern 為logstash-*

• 界面展示

  
  
  Paste_Image.png

8. 遇到錯誤

• 使用rpm安裝logstash時遇到如下錯誤，改成tar.gz解壓縮則是正常的。
  error: unpacking of archive failed on file /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.0.4-java/vendor/GeoLite2-City.mmdb;58776068: cpio: read failed - No such file or directory

• 在docker中啟動elasticsearch后，容器內可訪問，但用ip:port無法訪問。
  ERROR: bootstrap checks failed
  max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

• 在使用IP：PORT無法訪問時 需要將
  server.host: "localhost" 改成server.host: 0.0.0.0