MySQL MHA搭建

MySQL MHA

架構(gòu)介紹：MHA由兩部分組成MHA Manager（管理節(jié)點(diǎn)）和MHA Node（數(shù)據(jù)節(jié)點(diǎn)），MHA Node運(yùn)行在每臺MySQL服務(wù)器上，MHA Manager會定時(shí)探測集群中的master節(jié)點(diǎn)，當(dāng)master出現(xiàn)故障時(shí)，它可以自動將最新數(shù)據(jù)的slave提升為新的master，然后將所有其他的slave重新指向新的master

MHA的隱患：在MHA自動故障切換的過程中，MHA試圖從宕掉的主服務(wù)器上保存二進(jìn)制日志，最大程度保證數(shù)據(jù)的不丟失，存在的問題是，如果主服務(wù)器硬件故障宕機(jī)或無法通過SSH訪問，MHA沒有辦法保存二進(jìn)制日志，只能進(jìn)行故障轉(zhuǎn)移而可能丟失最新數(shù)據(jù)

工作原理總結(jié)為以下幾條:

????1.從宕機(jī)崩潰的master保存二進(jìn)制日志事件（binlog events）;

????2.識別含有最新更新的slave;

????3.應(yīng)用差異的中繼日志(relay log) 到其他slave;

????4.應(yīng)用從master保存的二進(jìn)制日志事件(binlog events);

????5.提升一個slave為新master;

????6.使用其他的slave連接新的master進(jìn)行復(fù)制。


1、安裝mysql：


????1.1 添加環(huán)境變量

????????????vim /etc/profile

????????????????export PATH=$PATH:/usr/local/mysql/bin

????????????source /etc/profile

????1.2????解壓tar包

????????????tar -xf mysql-5.7.22-linux-glibc2.12-x86_64.tar.gz

????????????mv mysql-5.7.22-linux-glibc2.12-x86_64 mysql

????????????scp -r /usr/local/mysql slave1:/usr/local/mysql

????????????scp -r /usr/local/mysql slave2:/usr/local/mysql

????????????scp /etc/my.cnf slave1:/etc/

????????????scp /etc/my.cnf slave2:/etc/

????????????所有節(jié)點(diǎn)my.cnf的server-id必須唯一

????1.3????創(chuàng)建用戶，目錄，授權(quán)，初始化,啟動（3臺執(zhí)行）

????????????useradd mysql

????????????mkdir -p /home/mysql3306/{mysql3306,logs}

????????????chown mysql:mysql /home/mysql3306 -R

????????????chown mysql:mysql /usr/local/mysql -R

????????????mysqld --defaults-file=/etc/my.cnf --initialize-insecure --datadir=/home/mysql3306/mysql3306 --basedir=/usr/local/mysql --user=mysql

????????????mysqld_safe --user=mysql &

2、配置主從

????2.1 在master上建立帳戶并授權(quán)slave:

????????????grant REPLICATION CLIENT,REPLICATION SLAVE on *.* to rep@'192.168.111.129' identified by '123456';

????????????grant REPLICATION CLIENT,REPLICATION SLAVE on *.* to rep@'192.168.111.130' identified by '123456';

????????????flush privileges;

????2.2 查看master狀態(tài),獲取binlog文件和pos點(diǎn)

????????????mysql> show master status;

????2.3 slave1、slave2設(shè)置需要同步的主庫

????????????change master to master_host='192.168.111.128',master_user='rep',master_password='123456', master_log_file='mysql-bin.000002',master_log_pos=1229,MASTER_PORT=3306;

????????????flush privileges;

????????????start slave;

????2.4 查看從服務(wù)器復(fù)制狀態(tài)

????????????show slave status\G


????2.5????兩臺slave服務(wù)器設(shè)置read_only（從庫對外提供讀服務(wù)，之所以沒有寫進(jìn)配置文件，是因?yàn)殡S時(shí)slave會提升為master）

????????????mysql -uroot -e "set global read_only=1"


????2.6????所有節(jié)點(diǎn)創(chuàng)建manager所需的監(jiān)控用戶

????????????grant all privileges on *.* to 'rep'@'192.168.111.%' identified??by '123456';

3、搭建MHA

????3.1????配置集群內(nèi)時(shí)間同步、ssh免密碼登陸

????3.2 MHA????node節(jié)點(diǎn)安裝


????????????yum install -y perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker perl-DBD-MySQL perl-devel perl-CPAN

????????????mkdir -p /etc/mha??##創(chuàng)建安裝目錄

????????????tar -xf mha4mysql-node-0.57.tar.gz -C /etc/mha/

????????????mv /etc/mha/mha4mysql-node-0.57 /etc/mha/node

????????????cd /etc/mha/node

????????????perl Makefile.PL

????????????make && make install


????3.3 MHA manager節(jié)點(diǎn)安裝????

????????????yum install -y perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-Time-HiRes

????????????tar -xf mha4mysql-manager-0.57.tar.gz -C /etc/mha/

????????????mv /etc/mha/mha4mysql-manager-0.57 /etc/mha/manager

????????????cd /etc/mha/manager

????????????perl Makefile.PL

????????????make && make install

????3.4 配置MHA


????????????修改manager配置文件

????????????mkdir /etc/mha/app1??##創(chuàng)建manager工作目錄

????????????cp /etc/mha/manager/samples/conf/app1.cnf /etc/mha/

????????????vim /etc/mha/app1.cnf

????????????????[server default]

????????????????manager_workdir=/etc/mha/app1????????????#MHA工作路徑

????????????????manager_log=/etc/mha/app1/manager.log????????#MHA日志路徑

????????????????master_binlog_dir="/home/mysql3306/mysql3306"????????#MHA node端的binlog路徑,也就是mysql的數(shù)據(jù)目錄

????????????????remote_workdir=/etc/mha/app1????????#遠(yuǎn)端mysql在發(fā)生切換時(shí)binlog的保存位置

????????????????master_ip_failover_script=/etc/mha/master_ip_failover????????#自動failover時(shí)候的切換腳本

????????????????master_ip_online_change_script=/etc/mha/master_ip_online_change????????#手動切換腳本

????????????????report_script=/etc/mha/send_report????????#發(fā)生切換后報(bào)警腳本

????????????????user=rep????????#監(jiān)控用戶

????????????????password=123456????????#監(jiān)控用戶密碼

????????????????repl_user=rep????????#復(fù)制用戶

????????????????repl_password=123456????????#復(fù)制用戶密碼

????????????????ping_interval=1????????????#MHA manager的檢測時(shí)間間隔(1秒)

????????????????secondary_check_script= masterha_secondary_check -s slave1 -s mastre --user=rep --master_host=master --master_ip=192.168.111.128 --master_port=3306????????#MHA檢測到master出現(xiàn)問題,Manager會嘗試從slave1登陸到master

????????????????[server1]

????????????????hostname=192.168.111.128

????????????????port=3306

????????????????ssh_port=22

????????????????[server2]

????????????????hostname=192.168.111.129

????????????????port=3306

????????????????candidate_master=1????????#備用主,如果主庫出問題,此庫將提升為主庫,即使這個庫不是集群中事件最新的slave

????????????????ssh_port=22

????????????????check_repl_delay=0????????#默認(rèn)情況下,一個slave落后于master 100M的relay log，MHA將不會選擇該slave為一個新的master,設(shè)置為0,MHA觸發(fā)切換在選擇一個新的master的時(shí)候?qū)雎詮?fù)制延時(shí),這個參數(shù)對于設(shè)置了candidate_master=1的主機(jī)非常有用，因?yàn)檫@個候選主在切換的過程中一定是新的master

????????????????[server3]

????????????????hostname=192.168.111.130

????????????????port=3306

????????????????no_master=1

????????????????ssh_port=22


????3.5????設(shè)置slave節(jié)點(diǎn)relay log清除方式;建立硬連接????


????????????MHA發(fā)生切換工程中,從庫恢復(fù)依賴于relay log,mysql默認(rèn)情況下,從庫應(yīng)用完就會自動清除relay log,因此將其設(shè)置為OFF,采用手動清理方式。


????????????????mysql -uroot -p123456 -e "set global relay_log_purge=0"????????????


????????????定期刪除relay log可能會出現(xiàn)復(fù)制延遲的問題,所以建立relay log日志硬連接,因?yàn)閘inux系統(tǒng)中通過硬連接刪除大文件速度快。


????????????????mkdir /home/mysql3306/logs1

????????????????ln /home/mysql3306/logs/mysql-relay* /home/mysql3306/logs1



????3.6????編寫定期清理relay log腳本,結(jié)合定時(shí)任務(wù)清理(slave1、slave2操作)????

????????????vim /etc/mha/purge_relay_log.sh

????????????????#!/bin/bash

????????????????user=root

????????????????passwd=123456

????????????????port=3306

????????????????log_dir='/home/mysql3306/logs/'

????????????????work_dir='/home/mysql3306/logs1'

????????????????purge='/usr/local/bin/purge_relay_logs'

????????????????if [ ! -d $log_dir ]

????????????????then

???????????????????mkdir $log_dir -p

????????????????fi

????????????????$purge --user=$user --password=$passwd --disable_relay_log_purge --port=$port --host=localhost --workdir=$work_dir >> $log_dir/purge_relay_logs.log 2>&1




????????????參數(shù)說明：

????????????????--work_dir：指定創(chuàng)建relay log的硬鏈接的位置

????????????????--disable_relay_log_purge ：默認(rèn)情況下，如果relay_log_purge=1，腳本會什么都不清理，自動退出，通過設(shè)定這個參數(shù)，當(dāng)relay_log_purge=1的情況下會將relay_log_purge設(shè)置為0。清理relay log之后，最后將參數(shù)設(shè)置為OFF。


????????????此處有幾個小細(xì)節(jié)

????????????????purge_relay_logs腳本中定義了的sock文件位置/var/lib/mysql/mysql.sock，可以做個軟鏈

????????????????????ln -s /tmp/mysql3306.sock /var/lib/mysql/mysql.sock

????????????????purge_relay_logs需要--user=root --host=localhost 沒有權(quán)限的，需要設(shè)置


????????????沒問題了，可以先測試下：


????????????????purge_relay_logs --user=root --host=localhost??--port=3306 --password=123456 -disable_relay_log_purge --workdir=/home/mysql3306/logs/


????????????出現(xiàn)這個說明測試通過：2018-07-04 05:22:21: All relay log purging operations succeeded.


????????????添加定時(shí)任務(wù)

????????????????crontab -e

????????????????0 0 */3 * * sh /etc/auto_clean_relay_log.sh????


????3.7????創(chuàng)建自動切換腳本

????????????vim /etc/mha/master_ip_failover

????????????????#!/usr/bin/env perl

????????????????use strict;

????????????????use warnings FATAL => 'all';

????????????????use Getopt::Long;

????????????????my (

????????????????$command, $ssh_user, $orig_master_host, $orig_master_ip,

????????????????$orig_master_port, $new_master_host, $new_master_ip, $new_master_port

????????????????);

????????????????my $vip = '192.168.111.111/24';

????????????????my $key = '0';

????????????????my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip";

????????????????my $ssh_stop_vip = "/sbin/ifconfig eth0:$key down";

????????????????GetOptions(

????????????????'command=s' => \$command,

????????????????'ssh_user=s' => \$ssh_user,

????????????????'orig_master_host=s' => \$orig_master_host,

????????????????'orig_master_ip=s' => \$orig_master_ip,

????????????????'orig_master_port=i' => \$orig_master_port,

????????????????'new_master_host=s' => \$new_master_host,

????????????????'new_master_ip=s' => \$new_master_ip,

????????????????'new_master_port=i' => \$new_master_port,

????????????????);

????????????????exit &main();

????????????????sub main {

????????????????print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

????????????????if ( $command eq "stop" || $command eq "stopssh" ) {

????????????????my $exit_code = 1;

????????????????eval {

????????????????print "Disabling the VIP on old master: $orig_master_host \n";

????????????????&stop_vip();

????????????????$exit_code = 0;

????????????????};

????????????????if ($@) {

????????????????warn "Got Error: $@\n";

????????????????exit $exit_code;

????????????????}

????????????????exit $exit_code;

????????????????}

????????????????elsif ( $command eq "start" ) {

????????????????my $exit_code = 10;

????????????????eval {

????????????????print "Enabling the VIP - $vip on the new master - $new_master_host \n";

????????????????&start_vip();

????????????????$exit_code = 0;

????????????????};

????????????????if ($@) {

????????????????warn $@;

????????????????exit $exit_code;

????????????????}

????????????????exit $exit_code;

????????????????}

????????????????elsif ( $command eq "status" ) {

????????????????print "Checking the Status of the script.. OK \n";

????????????????exit 0;

????????????????}

????????????????else {

????????????????&usage();

????????????????exit 1;

????????????????}

????????????????}

????????????????sub start_vip() {

????????????????`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;

????????????????}

????????????????sub stop_vip() {

????????????????return 0 unless ($ssh_user);

????????????????`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;

????????????????}

????????????????sub usage {

????????????????print

????????????????"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip

????????????????--orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";

????????????????}


????3.8????創(chuàng)建手動切換腳本

????????????vim /etc/mha/master_ip_online_change

????????????????#!/usr/bin/env perl

????????????????use strict;

????????????????use warnings FATAL =>'all';

????????????????use Getopt::Long;

????????????????my $vip = '192.168.111.111/24'; # Virtual IP

????????????????my $key = "0";

????????????????my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip";

????????????????my $ssh_stop_vip = "/sbin/ifconfig eth0:$key down";

????????????????my $exit_code = 0;

????????????????my (

????????????????$command, $orig_master_is_new_slave, $orig_master_host,

????????????????$orig_master_ip, $orig_master_port, $orig_master_user,

????????????????$orig_master_password, $orig_master_ssh_user, $new_master_host,

????????????????$new_master_ip, $new_master_port, $new_master_user,

????????????????$new_master_password, $new_master_ssh_user,

????????????????);

????????????????GetOptions(

????????????????'command=s' => \$command,

????????????????'orig_master_is_new_slave' => \$orig_master_is_new_slave,

????????????????'orig_master_host=s' => \$orig_master_host,

????????????????'orig_master_ip=s' => \$orig_master_ip,

????????????????'orig_master_port=i' => \$orig_master_port,

????????????????'orig_master_user=s' => \$orig_master_user,

????????????????'orig_master_password=s' => \$orig_master_password,

????????????????'orig_master_ssh_user=s' => \$orig_master_ssh_user,

????????????????'new_master_host=s' => \$new_master_host,

????????????????'new_master_ip=s' => \$new_master_ip,

????????????????'new_master_port=i' => \$new_master_port,

????????????????'new_master_user=s' => \$new_master_user,

????????????????'new_master_password=s' => \$new_master_password,

????????????????'new_master_ssh_user=s' => \$new_master_ssh_user,

????????????????);

????????????????exit &main();

????????????????sub main {

????????????????#print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

????????????????if ( $command eq "stop" || $command eq "stopssh" ) {

????????????????# $orig_master_host, $orig_master_ip, $orig_master_port are passed.

????????????????# If you manage master ip address at global catalog database,

????????????????# invalidate orig_master_ip here.

????????????????my $exit_code = 1;

????????????????eval {

????????????????print "\n\n\n***************************************************************\n";

????????????????print "Disabling the VIP - $vip on old master: $orig_master_host\n";

????????????????print "***************************************************************\n\n\n\n";

????????????????&stop_vip();

????????????????$exit_code = 0;

????????????????};

????????????????if ($@) {

????????????????warn "Got Error: $@\n";

????????????????exit $exit_code;

????????????????}

????????????????exit $exit_code;

????????????????}

????????????????elsif ( $command eq "start" ) {

????????????????# all arguments are passed.

????????????????# If you manage master ip address at global catalog database,

????????????????# activate new_master_ip here.

????????????????# You can also grant write access (create user, set read_only=0, etc) here.

????????????????my $exit_code = 10;

????????????????eval {

????????????????print "\n\n\n***************************************************************\n";

????????????????print "Enabling the VIP - $vip on new master: $new_master_host \n";

????????????????print "***************************************************************\n\n\n\n";

????????????????&start_vip();

????????????????$exit_code = 0;

????????????????};

????????????????if ($@) {

????????????????warn $@;

????????????????exit $exit_code;

????????????????}

????????????????exit $exit_code;

????????????????}

????????????????elsif ( $command eq "status" ) {

????????????????print "Checking the Status of the script.. OK \n";

????????????????`ssh $orig_master_ssh_user\@$orig_master_host \" $ssh_start_vip \"`;

????????????????exit 0;

????????????????}

????????????????else {

????????????????&usage();

????????????????exit 1;

????????????????}

????????????????}

????????????????# A simple system call that enable the VIP on the new master

????????????????sub start_vip() {

????????????????`ssh $new_master_ssh_user\@$new_master_host \" $ssh_start_vip \"`;

????????????????}

????????????????# A simple system call that disable the VIP on the old_master

????????????????sub stop_vip() {

????????????????`ssh $orig_master_ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;

????????????????}

????????????????sub usage {

????????????????print

????????????????"Usage: master_ip_failover -command=start|stop|stopssh|status -orig_master_host=host -orig_master_ip=ip -

????????????????orig_master_port=po

????????????????rt -new_master_host=host -new_master_ip=ip -new_master_port=port\n";

????????????????}????


????3.9編寫切換節(jié)點(diǎn)監(jiān)控報(bào)警腳本

????????????vim /etc/mha/send_report

????????????????#!/usr/bin/perl

????????????????use strict;

????????????????use warnings FATAL => 'all';

????????????????use Mail::Sender;

????????????????use Getopt::Long;

????????????????#new_master_host and new_slave_hosts are set only when recovering master succeeded

????????????????my ( $dead_master_host, $new_master_host, $new_slave_hosts, $subject, $body );

my $smtp='smtp.163.com';

my $mail_from='xxxxx@163.com';

my $mail_user='xxxxx@163.com';

????????????????my $mail_pass='xxxxx';

my $mail_to=['xxxxx@139.com'];

????????????????GetOptions(

??????????????????'orig_master_host=s' => \$dead_master_host,

??????????????????'new_master_host=s'??=> \$new_master_host,

??????????????????'new_slave_hosts=s'??=> \$new_slave_hosts,

??????????????????'subject=s'??????????=> \$subject,

??????????????????'body=s'?????????????=> \$body,

????????????????);

????????????????mailToContacts($smtp,$mail_from,$mail_user,$mail_pass,$mail_to,$subject,$body);

????????????????sub mailToContacts {

????????????????????my ( $smtp, $mail_from, $user, $passwd, $mail_to, $subject, $msg ) = @_;

????????????????????open my $DEBUG, "> /tmp/monitormail.log"

????????????????????????or die "Can't open the debug??????file:$!\n";

????????????????????my $sender = new Mail::Sender {

????????????????????????ctype???????=> 'text/plain; charset=utf-8',

????????????????????????encoding????=> 'utf-8',

????????????????????????smtp????????=> $smtp,

????????????????????????from????????=> $mail_from,

????????????????????????auth????????=> 'LOGIN',

????????????????????????TLS_allowed => '0',

????????????????????????authid??????=> $user,

????????????????????????authpwd?????=> $passwd,

????????????????????????to??????????=> $mail_to,

????????????????????????subject?????=> $subject,

????????????????????????debug???????=> $DEBUG

????????????????????};

????????????????????$sender->MailMsg(

????????????????????????{???msg???=> $msg,

????????????????????????????debug => $DEBUG

????????????????????????}

????????????????????) or print $Mail::Sender::Error;

????????????????????return 1;

????????????????}

????????????????# Do whatever you want here

????????????????exit 0;????


????????????????腳本需要修改的地方

my $smtp='smtp.163.com';???????????????????## 提供smtp服務(wù)的服務(wù)商地址，通常為smtp.(qq.163.139.)com

my $mail_from='xxxxx@163.com';?????????????????## 發(fā)送郵件的郵箱

my $mail_user='xxxxx@163.com';???????????????## 同上

????????????????????my $mail_pass='xxxxx';???????????????????????## 郵箱授權(quán)碼，郵箱開啟pop3/smtp時(shí)，一般會讓你設(shè)置密碼

my $mail_to=['xxxxx@139.com'];???????????????## 接收郵件的郵箱,139為移動的短信郵箱,很方便,直接短信接收信息



????????????給其執(zhí)行權(quán)限

????????????????chmod +x /etc/mha/master_ip_failover

????????????????chmod +x /etc/mha/master_ip_online_change

????????????????chmod +x /etc/mha/send_report


????3.10manager檢查ssh是否成功????????


????????????/etc/mha/manager/bin/masterha_check_ssh --conf=/etc/mha/app1.cnf


????3.11manager檢查復(fù)制狀態(tài)

????????????所有節(jié)點(diǎn)創(chuàng)建軟鏈

????????????????ln -s /usr/local/mysql/bin/mysqlbinlog /usr/bin/mysqlbinlog

????????????????ln -s /usr/local/mysql/bin/mysql /usr/bin/mysql


????????/etc/mha/manager/bin/masterha_check_repl --conf=/etc/mha/app1.cnf


????3.12為master添加vip

????????????ifconfig ens33:0 192.168.111.111


????3.13manager節(jié)點(diǎn)啟動mha

????????????nohup /etc/mha/manager/bin/masterha_manager --conf=/etc/mha/app1.cnf --ignore_last_failover >/tmp/mha_manager.log < /dev/null 2>&1 &


????3.14檢查mha狀態(tài)

????????/etc/mha/manager/bin/masterha_check_status --conf=/etc/mha/app1.cnf


????3.15測試????


????????實(shí)驗(yàn)一：測試自動Failover


????????????1.在slave1 上我先停掉IO線程，模擬主從延遲

????????????????stop slave io_thread;


????????????2.master庫導(dǎo)入一張表(數(shù)據(jù)量盡量大點(diǎn),建議10W+以上數(shù)據(jù))

????????????????這時(shí)候slave2一直在同步數(shù)據(jù)


????????????3.slave1開啟IO線程

????????????????start slave io_thread;


????????????4.停掉master mysql

????????????????實(shí)驗(yàn)使用pkill mysql(生產(chǎn)禁用)


????????????5.查看manager日志,可以看出master已經(jīng)換了

????????????????tail -300f /etc/mha/app1/manager.log


????????????6.在新的master上可以看到落后的數(shù)據(jù)也已經(jīng)同步過來了


????????????7.查看Vip飄逸情況,vip是否到了slave1這臺主機(jī)


????????實(shí)驗(yàn)二：手動Failover測試


????????????注意,執(zhí)行手動Failover時(shí),MHA manager必須沒有運(yùn)行,否則,manager會掛掉


????????????1.停止manager和master的mysql

????????????????/etc/mha/manager/bin/masterha_stop --conf=/etc/mha/app1.cnf

????????????????實(shí)驗(yàn)使用pkill mysql(生產(chǎn)禁用)


????????????2.執(zhí)行manager上的腳本master_ip_online_change

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
平臺聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡書系信息發(fā)布平臺，僅提供信息存儲服務(wù)。
  • 人面猴
    序言：七十年代末，一起剝皮案震驚了整個濱河市，隨后出現(xiàn)的幾起案子，更是在濱河造成了極大的恐慌，老刑警劉巖，帶你破解...
    沈念sama閱讀 228,786評論 6贊 534
  • 死咒
    序言：濱河連續(xù)發(fā)生了三起死亡事件，死亡現(xiàn)場離奇詭異，居然都是意外死亡，警方通過查閱死者的電腦和手機(jī)，發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 98,656評論 3贊 419
  • 救了他兩次的神仙讓他今天三更去死
    文/潘曉璐 我一進(jìn)店門，熙熙樓的掌柜王于貴愁眉苦臉地迎上來，“玉大人，你說我怎么就攤上這事。” “怎么了？”我有些...
    開封第一講書人閱讀 176,697評論 0贊 379
  • 道士緝兇錄：失蹤的賣姜人
    文/不壞的土叔 我叫張陵，是天一觀的道長。 經(jīng)常有香客問我，道長，這世上最難降的妖魔是什么？ 我笑而不...
    開封第一講書人閱讀 63,098評論 1贊 314
  • ?港島之戀（遺憾婚禮）
    正文 為了忘掉前任，我火速辦了婚禮，結(jié)果婚禮上，老公的妹妹穿的比我還像新娘。我一直安慰自己，他們只是感情好，可當(dāng)我...
    茶點(diǎn)故事閱讀 71,855評論 6贊 410
  • 惡毒庶女頂嫁案：這布局不是一般人想出來的
    文/花漫 我一把揭開白布。 她就那樣靜靜地躺著，像睡著了一般。 火紅的嫁衣襯著肌膚如雪。 梳的紋絲不亂的頭發(fā)上，一...
    開封第一講書人閱讀 55,254評論 1贊 324
  • 城市分裂傳說
    那天，我揣著相機(jī)與錄音，去河邊找鬼。 笑死，一個胖子當(dāng)著我的面吹牛，可吹牛的內(nèi)容都是我干的。 我是一名探鬼主播，決...
    沈念sama閱讀 43,322評論 3贊 442
  • 雙鴛鴦連環(huán)套：你想象不到人心有多黑
    文/蒼蘭香墨 我猛地睜開眼，長吁一口氣：“原來是場噩夢啊……” “哼！你這毒婦竟也來了？” 一聲冷哼從身側(cè)響起，我...
    開封第一講書人閱讀 42,473評論 0贊 289
  • 萬榮殺人案實(shí)錄
    序言：老撾萬榮一對情侶失蹤，失蹤者是張志新（化名）和其女友劉穎，沒想到半個月后，有當(dāng)?shù)厝嗽跇淞掷锇l(fā)現(xiàn)了一具尸體，經(jīng)...
    沈念sama閱讀 49,014評論 1贊 335
  • ?護(hù)林員之死
    正文 獨(dú)居荒郊野嶺守林人離奇死亡，尸身上長有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點(diǎn)故事閱讀 40,833評論 3贊 355
  • ?白月光啟示錄
    正文 我和宋清朗相戀三年，在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了。 大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片。...
    茶點(diǎn)故事閱讀 43,016評論 1贊 371
  • 活死人
    序言：一個原本活蹦亂跳的男人離奇死亡，死狀恐怖，靈堂內(nèi)的尸體忽然破棺而出，到底是詐尸還是另有隱情，我是刑警寧澤，帶...
    沈念sama閱讀 38,568評論 5贊 362
  • ?日本核電站爆炸內(nèi)幕
    正文 年R本政府宣布，位于F島的核電站，受9級特大地震影響，放射性物質(zhì)發(fā)生泄漏。R本人自食惡果不足惜，卻給世界環(huán)境...
    茶點(diǎn)故事閱讀 44,273評論 3贊 347
  • 男人毒藥：我在死后第九天來索命
    文/蒙蒙 一、第九天 我趴在偏房一處隱蔽的房頂上張望。 院中可真熱鬧，春花似錦、人聲如沸。這莊子的主人今日做“春日...
    開封第一講書人閱讀 34,680評論 0贊 26
  • 一樁弒父案，背后竟有這般陰謀
    文/蒼蘭香墨 我抬頭看了看天上的太陽。三九已至，卻和暖如春，著一層夾襖步出監(jiān)牢的瞬間，已是汗流浹背。 一陣腳步聲響...
    開封第一講書人閱讀 35,946評論 1贊 288
  • 情欲美人皮
    我被黑心中介騙來泰國打工， 沒想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留，地道東北人。 一個月前我還...
    沈念sama閱讀 51,730評論 3贊 393
  • 代替公主和親
    正文 我出身青樓，卻偏偏與公主長得像，于是被迫代替她去往敵國和親。 傳聞我的和親對象是個殘疾皇子，可洞房花燭夜當(dāng)晚...
    茶點(diǎn)故事閱讀 48,006評論 2贊 374