Elasticsearch

Elasticsearch是一個基于lucene的全文搜索引擎

1.安裝

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.3.0.tar.gz

tar -xzvf elasticsearch-5.3.0.tar.gz

mv elasticsearch-5.3.0  /usr/share/elasticsearch-5.3.0 

2. 配置

修改配置文件

path.data: /var/lib/elasticsearch //數據存放的路徑
path.logs: /var/log/elasticsearch //log的路徑
node.name: wddlc //節點的名字
network.host: _網卡編號_  
http.port: 9206  //監聽的端口號

配置文件：/usr/share/elasticsearch-5.3.0/config/elasticsearch.yml

如果需要的話修改jvm的內存參數

-Xms1g   初始內存大小
-Xmx1g   最大內存大小

配置文件：/usr/share/elasticsearch-5.3.0/config/jvm.options

3. 啟動

nohup /usr/share/elasticsearch-5.3.0/bin/elasticsearch -Epath.conf=/usr/share/elasticsearch-5.3.0/config/ > /dev/null 2>&1 &

ps -ef | grep elas  //關閉
kill pid

檢測啟動是否成功.png

4.一個小問題

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

這是因為系統進程最大虛擬內存映射的數量不足，用下面的命令設置一下就可以了

sysctl -w vm.max_map_count=262144

5.監控插件

git clone git://github.com/mobz/elasticsearch-head.git

cd elasticsearch-head

npm install

npm run start

open
 [http://localhost:9100/](http://localhost:9100/)

注意：使用這個插件的時候，要打開es的跨域

http.cors.enabled: true
http.cors.allow-origin: "*"

logstash

logstash是一個開源的數據收集引擎，負責從多個數據源匯總數據、過濾、并輸出到指定的輸出中

1.安裝

wget wget https://artifacts.elastic.co/downloads/logstash/logstash-5.3.0.tar.gz

tar -xzvf logstash-5.3.0.tar.gz

mv logstash-5.3.0  /usr/share/logstash-5.3.0

2.配置

input {
    beats {
        #  監聽filebeat的輸入
        port => 5043
    }
}
filter {
  # 刪除無用的key
    if ([type] == "log") {
        ruby {
            code => "
                event.to_hash.each { |k,v|
                    if (!['json', 'type'].include?(k))
                        event.remove(k)
                    end
                }
            "
        }
# 根據ip，補充地理位置信息
        geoip {
            source => "[json][ip]"
        }
    }
}
output {
    if ([type] == "log") {
# 輸出到標準輸出
        stdout { codec => rubydebug}
# 輸出到剛剛配置好的es里
        elasticsearch {
            hosts => ["172.16.3.2:9206"]
            index => "%{[json][model]}"
        }
    }
}

2.啟動

nohup /usr/share/logstash-5.3.0/bin/logstash -f /usr/share/logstash-5.3.0/config/first-pipeline.conf > /etc/logstash.log 2>&1 &

filebeat

1.安裝

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.3.0-x86_64.rpm
sudo rpm -vi filebeat-5.3.0-x86_64.rpm

2.配置

配置文件在：/etc/filebeat/filebeat.yml

# 設置輸入
 input_type: log
# 表示數據的是json
  json.key_under_root: true
# 監控這個路徑下的log
  paths:
    - /var/www/prometheus-library/logs/*.log

# 設置控制臺輸出，方便調試，正式環境可以關掉
output.console:
  pretty: true

# 輸出到logstash
output.logstash:
  hosts: ["172.16.3.2:5043"]

3.啟動

sudo /etc/init.d/filebeat start

kibana

1.安裝

wget https://artifacts.elastic.co/downloads/kibana/kibana-5.3.0-linux-x86_64.tar.gz

tar -xzvf kibana-5.3.0-linux-x86_64.tar.gz

mv kibana-5.3.0-linux-x86_64 /usr/share/kibana-5.3.0-linux-x86_64

2.配置

配置文件：/usr/share/kibana-5.3.0-linux-x86_64/config/kibana.yml

# 設置es的地址
server.host: "0.0.0.0"

# es請求的路徑
elasticsearch.url: "http://172.16.3.2:9206"

2.啟動

nohup /usr/share/kibana-5.3.0-linux-x86_64/bin/kibana > kibana.log 2>&1 &