一 前言
Bottled Water是Confluent公司開發的一款可以將postgresql數據庫轉換為kafka events的工具。主要用于監視PG的數據增刪改的變化，將變化的數據實時推送到kafka消息隊列，其他關心數據變化的平臺只要監聽kafka的消息，就能得到改變的數據。具體參考德哥的博客《實時數據交換平臺 - BottledWater-pg with confluent》，本文具體的安裝步驟也參考了《Bottled Water: 實時集成postgresql與kafka》。
二 安裝準備
安裝完成PostgreSQL，參考《Centos7安裝PostgreSQL9.6》；
安裝完成Kafka，參考《Kafka部署》；
此外還依賴avro-c，Jansson，libcurl，librdkafka
三 安裝步驟bottledwater-pg
3.1 安裝libjansson
wget http://www.digip.org/jansson/releases/jansson-2.9.tar.bz2
tar -jxvf jansson-2.9.tar.bz2
cd jansson-2.9
./configure --prefix=/home/free/jansson
make
make install
export PKG_CONFIG_PATH=/home/free/jansson/lib/pkgconfig:$PKG_CONFIG_PATH
pkg-config --cflags --libs jansson
-I/home/free/jansson/include -L/home/free/jansson/lib -ljansson
3.1 安裝avro
wget http://mirrors.hust.edu.cn/apache/avro/avro-1.8.1/avro-src-1.8.1.tar.gz
tar -zxvf avro-src-1.8.1.tar.gz
cd avro-src-1.8.1/lang/c
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/home/free/avro -DCMAKE_BUILD_TYPE=Release -DTHREADSAFE=true
make
make test
make install
export PKG_CONFIG_PATH=/home/free/avro/lib/pkgconfig:$PKG_CONFIG_PATH
在make這一步驟報錯如下：
opt/avro-src-1.8.1/lang/c/src/schema.c:31:21: fatal error: jansson.h: No such file or directory
#include "jansson.h"
^
compilation terminated.
make[2]: *** [src/CMakeFiles/avro-shared.dir/schema.o] Error 1
make[1]: *** [src/CMakeFiles/avro-shared.dir/all] Error 2
make: *** [all] Error 2
這一步驟應該是編譯缺少了頭文件，先加上去，然后重新編譯。
export C_INCLUDE_PATH=/home/free/jansson/include:$C_INCLUDE_PATH
3.3 安裝libcurl
wget https://curl.haxx.se/download/curl-7.54.0.tar.gz
tar -zxvf curl-7.54.0.tar.gz
cd curl-7.54.0
./configure --prefix=/home/free/curl
make
make install
export PKG_CONFIG_PATH=/home/free/curl/lib/pkgconfig:$PKG_CONFIG_PATH
3.4 安裝librdkafka
git clone https://github.com/edenhill/librdkafka
cd librdkafka
./configure --prefix=/home/free/librdkafka
#假如報錯如下：
#[root@bogon librdkafka]# ./configure --prefix=/home/freegis/librdkafka
#-bash: ./configure: Permission denied
#賦予執行權限
#[root@bogon librdkafka]# chmod 777 configure
make -j 32
make install
export PKG_CONFIG_PATH=/home/free/librdkafka/lib/pkgconfig:$PKG_CONFIG_PATH
在make的時候，64位的linux會報錯如下：
/bin/ld:librdkafka.lds:1: syntax errer in VERSION script
只需要在Makefile.config里面的WITH_LDS=y這一行注釋掉重新make就可以了。
3.5 安裝bottledwater-pg
在環境變量中新增如下依賴：
vi /etc/profile
#編輯內容如下
export CURLHOME=/home/freegis/curl
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$GCCHOME/bin:$CURLHOME/bin
# wq!保存
source /etc/profile
下載安裝文件并編譯部署：
git clone https://github.com/confluentinc/bottledwater-pg
cd bottledwater-pg
#啟用postgres的環境變量
source /home/postgres/.bashrc
make
make install
make可能報錯如下：
ake[1]: Entering directory `/opt/bottledwater-pg-master/kafka'
gcc bottledwater.c -c -std=c99 -D_POSIX_C_SOURCE=200809L -I../client -I../ext -I/home/postgres/include -I/home/postgres/include/server -I/home/freegis/avro/include -DAVRO_1_8 -Wall -Wmissing-prototypes -Wpointer-arith -Wendif-labels -Wmissing-format-attribute -Wformat-security -o bottledwater.o
In file included from table_mapper.h:4:0,
from json.h:5,
from bottledwater.c:2:
registry.h:4:32: fatal error: librdkafka/rdkafka.h: No such file or directory
#include <librdkafka/rdkafka.h>
^
compilation terminated.
這種情況仍然是頭文件找不到，為了麻煩，把用到的頭文件都指定。
export C_INCLUDE_PATH=/home/free/jansson/include:$C_INCLUDE_PATH
export C_INCLUDE_PATH=/home/free/avro/include:$C_INCLUDE_PATH
export C_INCLUDE_PATH=/home/free/librdkafka/include:$C_INCLUDE_PATH
export C_INCLUDE_PATH=/home/free/curl/include:$C_INCLUDE_PATH
繼續編譯后報錯：
/usr/bin/ld: cannot find -lrdkafka
詳細如下：
缺失依賴.png
建立依賴軟連接：
ln -s /home/free/librdkafka/lib/librdkafka.so /home/postgres/lib/librdkafka.so
檢查安裝情況
su - postgres
psql
create extension bottledwater;
如果創建擴展報錯如下：
mcsas=# create extension bottledwater;
ERROR: could not load library "/home/postgres/lib/bottledwater.so": libavro.so.23.0.0: cannot open shared object file: No such file or directory
查看下bottledwater.so的依賴：
[postgres@bogon lib]$ ldd bottledwater.so
linux-vdso.so.1 => (0x00007fffc41fe000)
libavro.so.23.0.0 => /home/freegis/avro/lib/libavro.so.23.0.0 (0x00007f1f26769000)
libc.so.6 => /lib64/libc.so.6 (0x00007f1f263a8000)
libjansson.so.4 => /lib64/libjansson.so.4 (0x00007f1f2619c000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1f25f7f000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1f26bba000)
發現所有的依賴都正確。
后經大神提示，重啟數據庫，然后執行擴展。
pg_ctl restart
psql
\c mcsas
create extension bottledwater;
CREATE EXTENSION
