MySQL數據同步方案實戰

準備

  • MySQL 5.6
  • Canal 1.0.24
  • Kafka 1.0.0
  • Zookeeper 3.4.11
  • Spring Boot
  • JDK 8
  • IntelliJ Idea 14.1.7

背景

在實際工作中，多個系統使用相同的用戶等業務數據是常有的事情。隨著需求的迭代式開發，系統的增加。系統的業務數據變更通知逐漸成為痛點，將數據變更逐一通知到其他系統在系統少的時候勉強可以接受，但是系統變多，需要通知的業務數據增多再使用此類的方式無疑是得不償失的，且極不易維護，耦合性太強，牽一發而動全身。此時，引入中間件無疑是一種明智的選擇。將系統從繁雜的通知中解脫出來，把通知這件事交給中間件來做。而這是基于業務操作最終會落實到數據庫中，這也是本文方案可行的前提。

架構圖

image

實現

Canal Server搭建

MySQL配置

  • 進入C:\Program Files\MySQL\MySQL Server 5.6目錄
  • 編輯my.ini文件（按需修改），打開mysql bin log功能，如下
[mysqld]
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M

# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin

# These are commonly set, remove the # and set as required.
basedir = C:\Program Files\MySQL\MySQL Server 5.6
datadir = C:\Program Files\MySQL\MySQL Server 5.6\data
# port = .....
# server_id = .....

log-bin=mysql-bin  #添加這一行即可
#選擇row模式
binlog-format=ROW
#配置mysql replaction需要定義，不能和canal的slaveId重復
server_id=1 
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M 

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
  • canal的原理是模擬自己為mysql slave，所以這里一定需要擁有做為mysql slave的相關權限（canal server配置需要用到此用戶）
CREATE USER canal IDENTIFIED BY 'canal';  
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;
  • 重啟mysql server
  • 查看bin log狀態
show binlog events;
image
#canal.properties
##配置zookeeper地址
canal.zkServers=127.0.0.1:2181

#################################################
#instance.properties
## mysql serverId
canal.instance.mysql.slaveId = 1234

# position info,需要改成自己的數據庫信息
canal.instance.master.address = 127.0.0.1:3306
canal.instance.master.journal.name = 
canal.instance.master.position = 
canal.instance.master.timestamp = 

#canal.instance.standby.address = 
#canal.instance.standby.journal.name =
#canal.instance.standby.position = 
#canal.instance.standby.timestamp = 

# username/password,需要改成自己的數據庫信息
canal.instance.dbUsername = canal
canal.instance.dbPassword = canal
canal.instance.defaultDatabaseName =
canal.instance.connectionCharset = UTF-8

# table regex
canal.instance.filter.regex = .*\\..*
# table black regex
canal.instance.filter.black.regex =  

#################################################

  • 啟動canal server

    • 進入D:\MySQL_Sync_Component\canal.deployer-1.0.24\bin目錄
    • 運行startup.bat

  • 查看canal server日志

    • D:\MySQL_Sync_Component\canal.deployer-1.0.24\logs\canal\canal.log
    • D:\MySQL_Sync_Component\canal.deployer-1.0.24\logs\example\example.log
canal.log

2017-11-24 17:55:32.550 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.
2017-11-24 17:55:32.770 [main] INFO  com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[192.168.191.1:11111]
2017-11-24 17:55:34.503 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......

example.log

2017-11-24 17:55:33.202 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2017-11-24 17:55:33.209 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
2017-11-24 17:55:33.343 [main] WARN  org.springframework.beans.TypeConverterDelegate - PropertyEditor [com.sun.beans.editors.EnumEditor] found through deprecated global PropertyEditorManager fallback - consider using a more isolated form of registration, e.g. on the BeanWrapper/BeanFactory!
2017-11-24 17:55:33.471 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example 
2017-11-24 17:55:33.647 [main] INFO  c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....
2017-11-24 17:55:33.896 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN  c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - prepare to find start position just show master status

Zookeeper Server搭建

  • 獲取并配置zookeeper（http://www.apache.org/dyn/closer.cgi/zookeeper/）
  • 復制D:\MySQL_Sync_Component\zookeeper\zookeeper-3.4.11\conf\zoo_sample.cfg并重命名為zoo.cfg,zookeeper讀取的是zoo.cfg這個配置文件
  • 啟動zookeeper，進入D:\MySQL_Sync_Component\zookeeper\zookeeper-3.4.11\bin，執行
zkServer.cmd

Kafka Server搭建

#server.properteis#
#配置zookeeper連接地址
zookeeper.connect=localhost:2181
  • 啟動Kafka服務，進入D:\MySQL_Sync_Component\kafka\kafka_2.12-1.0.0\bin\windows，執行
kafka-server-start.bat D:\MySQL_Sync_Component\kafka\kafka_2.12-1.0.0\bin\conf\server.properties
  • 創建topic，進入D:\MySQL_Sync_Component\kafka\kafka_2.12-1.0.0\bin\windows，執行
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testbinlog
  • 啟動一個kafka生產者，用于測試kafka是否啟動成功。執行
kafka-console-producer.bat --broker-list localhost:9092 --topic testbinlog
  • 啟動一個kafka消費者，接收生產者發送的消息。測試通過，consumer可以收到producer發送的消息。執行
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic testbinlog --from-beginning

創建Spring Boot應用

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

總結

如上，已經介紹了整個解決方案所需組件的配置及啟動方法。文中的解決方案用于業務數據通知或者緩存刷新等場景。本文除了提供一種業務場景的解決方案以外，更多的是希望讀者可以從中領會到各組件的設計思想及其優劣勢。除了可以熟練配置及使用各組件，這些組件的編碼思想及設計模式更是我輩學習的典范。對于各組件的使用，最好先去閱讀官方文檔，然后參考相關博客，再編碼實踐，最后閱讀下源碼，達到融會貫通的境界。
PS:
示例代碼github地址：
https://github.com/hxysea/mynote/tree/master/%E9%A1%B9%E7%9B%AE%E7%BB%8F%E9%AA%8C/Mysql%E6%95%B0%E6%8D%AE%E5%90%8C%E6%AD%A5%E8%A7%A3%E5%86%B3%E6%96%B9%E6%A1%88

參考文檔

https://github.com/alibaba/canal/wiki

http://kafka.apache.org/

https://docs.spring.io/spring-boot/docs/1.5.8.RELEASE/reference/htmlsingle/

最后編輯于 ：
?著作權歸作者所有,轉載或內容合作請聯系作者
平臺聲明：文章內容（如有圖片或視頻亦包括在內）由作者上傳并發布，文章內容僅代表作者本人觀點，簡書系信息發布平臺，僅提供信息存儲服務。

推薦閱讀更多精彩內容