1，目的
在了解Docker容器、鏡像和倉庫基本使用的情況下，可嘗試搭建Docker集群，進入Docker云計算時代。
通過本文的學習，可以了解學習Docker集群的搭建和使用，了解學習Docker的基本網絡配置。
2，Docker集群模塊劃分
本文將搭建一個簡單的Docker集群，包含三個模塊，分別是消息模塊、計算模塊和測試模塊。
消息模塊，docker_message
使用redis作為消息中間件，包含三個list：list_message_in、list_message_out和list_message_alive，只部署一份計算模塊，docker_worker
啟動Python程序，讀取docker_message的list_message_in，完成計算后，將結果輸出到list_message_out，可以啟動多份。另，每秒輸出心跳到list_message_alive（空閑時間）測試模塊，redis-cli或python程序
通過redis-cli或python程序測試Docker集群是否運行正常
3，消息模塊搭建
啟動消息模塊
# 創建網絡，docker有四種網絡類型（bridge、container、host和none），我們使用的是none，可以設置靜態IP
# docker network create --subnet=172.18.0.0/24 my_network
0b85ab8c7809886bb7fada8c4a8970b5692f11dbd950790f1e9b1e098691f114
# 啟動新容器，設置靜態IP
# docker run -it --name docker_message -h docker_message -p 26379:6379 --net my_network --ip 172.18.0.2 --add-host docker_message:172.18.0.2 learn/redis:v2 /bin/sh /etc/rc.local
參數說明
- -it，交互方式啟動
- --name docker_message，指定新容器的名稱是docker_message
- -h docker_message，指定新容器的主機名是docker_message
- -p 26379:6379，指定宿主機與docker容器的端口映射，宿主機的26379對應docker容器的6379
- --net my_network，指定網絡是新創建的網絡my_network，IP段是172.18.0.0/24
- --ip 172.18.0.2，指定容器的IP為靜態IP 172.18.0.2
- --add-host docker_message:172.18.0.2，增加容器的host配置，docker_message:172.18.0.2
- learn/redis:v2，指定啟動容器對應的鏡像是learn/redis:v2，可以是鏡像ID
- /bin/sh /etc/rc.local，指定容器啟動后，執行shell腳本是/etc/rc.local
進入到容器，安裝redis，并啟動測試
root@docker_message:/# apt-get install redis-server
root@docker_message:/# service redis-server start
Starting redis-server: redis-server.
root@docker_message:/# redis-cli -h docker_message
docker_message:6379> set a 00
OK
docker_message:6379> get a
"00"
測試Docker容器的網絡設置
- 通過宿主機和外網機器（非172.18.0.0/24網段）的機器，連接docker_message的redis服務，測試是否正常
# 宿主機的測試
kevin@apple:~$ redis-cli -h 172.18.0.2 -p 6379
172.17.0.2:6379> get a
(nil)
172.17.0.2:6379> keys *
(empty list or set)
172.17.0.2:6379> set a 11
OK
172.17.0.2:6379> get a
"11"
# 外網機器的測試
kevin@orange:~/docker$ redis-cli -h 192.168.1.89 -p 26379
192.168.1.89:26379> set a 22
OK
192.168.1.89:26379> get a
"22"
- 在docker_message查看網絡連接情況，發現6379端口有來自192.168.1.32（外網機器）和172.18.0.1（宿主機）的TCP連接，說明網絡設置OK
root@docker_message:/# netstat -pan
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.11:33007 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 30/sshd
tcp 0 0 172.18.0.2:6379 172.18.0.1:52440 ESTABLISHED -
tcp 0 0 172.18.0.2:6379 192.168.1.32:36922 ESTABLISHED -
tcp6 0 0 :::6379 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN 30/sshd
udp 0 0 127.0.0.11:60720 0.0.0.0:* -
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node PID/Program name Path
修改docker容器的開機腳本
# vi /etc/rc.local
####!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
service redis-server start
/bin/bash
exit 0
4，計算模塊搭建
啟動一個docker容器
# docker run -it --name docker_worker1 -h docker_worker1 --net my_network --add-host docker_message:172.18.0.2 learn/redis:v2 /bin/bash
參數說明
- --net my_network，指定容器使用的網絡是my_network，這樣可以與docker_message內網通信
- --add-host docker_message:172.18.0.2，增加容器的host配置，docker_message:172.18.0.2，可以通過docker_message連接消息模塊
- 備注：可以通過-e docker_message=172.18.0.2增加環境變量
網絡測試
root@docker_worker1:/# ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:ac:12:00:03
inet addr:172.18.0.3 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:12 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1856 (1.8 KB) TX bytes:0 (0.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
root@docker_worker1:/# redis-cli -h docker_message -p 6379
docker_message:6379> set a 22
OK
docker_message:6379> get a
"22"
編寫Python腳本
- 安裝python包
# apt-get install python-redis
- 創建python腳本
root@docker_worker1:/home/visual/python# vi /home/visual/python/worker.py
#coding:utf8
import redis, socket, time
# 計算函數
def worker():
# 連接redis
# docker_message = "192.168.1.89"
docker_message = "docker_message"
r = redis.Redis(host=docker_message, port=6379, db=0)
# 從list_message_in獲取數據，計算后寫入到list_message_out
list_message_in = "list_message_in"
list_message_out = "list_message_out"
list_message_alive = "list_message_alive"
hostname = socket.gethostname()
while(True):
str_message = r.rpop(list_message_in)
# 如果獲取的消息為空，則休息1秒再看是否有數據
if str_message == None:
if r.llen(list_message_alive) >= 100:
r.delete(list_message_alive)
str_alive_message = hostname + " is alive at " + time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(time.time()))
r.lpush(list_message_alive, str_alive_message)
time.sleep(1)
continue
# 完成表達式的計算，并返回結果
try:
str_message = str_message + "=" + str(eval(str_message))
except Exception,e:
print str_message,e
str_response = "worker %s return %s" % ( hostname, str_message )
r.lpush(list_message_out, str_response)
# print str_response
if __name__ == "__main__":
worker()
- 容器中啟動Python腳本
root@docker_worker1:/home/visual/python# python worker.py 2>&1 &
- 測試Python運行正常
root@apple:/home/kevin/docker/docker_cluster# redis-cli -p 26379
127.0.0.1:26379> keys *
1) "list_message_out"
2) "list_message_alive"
127.0.0.1:26379> llen list_message_alive
(integer) 67
127.0.0.1:26379> llen list_message_alive
(integer) 68
127.0.0.1:26379> llen list_message_alive
(integer) 69
127.0.0.1:26379> llen list_message_alive
(integer) 70
127.0.0.1:26379> lpush list_message_in "1+2-3+3*3"
(integer) 1
127.0.0.1:26379> lrange list_message_out 0 3
1) "worker docker_worker1 return 1+2-3+3*3=9"
2) "1+2+3*2=9"
3) "1+2+3*2"
4) "None"
添加開機啟動
root@docker_worker1:/# cat /etc/rc.local
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
# service ssh start
# service redis-server start
python /home/visual/python/worker.py 2>&1 &
/bin/bash
exit 0
重新打包生成鏡像learn/worker:v1，可還原部署多份容器
kevin@apple:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a5e5efb3f995 learn/redis:v2 "/bin/bash" 2 hours ago Exited (0) 14 seconds ago docker_worker1
ad7291dea597 learn/redis:v2 "/bin/sh /etc/rc.l..." 2 hours ago Up 2 hours 0.0.0.0:26379->6379/tcp docker_message
cbbbe7a5d47a learn/nginx:v2 "/bin/sh /etc/rc.l..." 6 hours ago Exited (255) 3 hours ago 0.0.0.0:8000->80/tcp, 0.0.0.0:8001->81/tcp nginx_test
93a1b9d39683 ubuntu "bash" 2 days ago Exited (0) 7 hours ago zealous_noether
abdc084f9821 hello-world "/hello" 2 days ago Exited (0) 26 hours ago sleepy_clarke
kevin@apple:~$ docker commit a5e5efb3f995 learn/worker:v1
sha256:abb80d7a396cca5d99d3253a85a9629413a962e1dd7a18c77b53ce97d1cbd0f9
kevin@apple:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
learn/worker v1 abb80d7a396c 16 seconds ago 325MB
learn/redis v2 9385e5566f04 2 hours ago 324MB
learn/nginx v2 3393d626158d 6 hours ago 370MB
learn/nginx v1 ab92edd21696 8 hours ago 370MB
learn/redis v1 619ec66bf6dd 18 hours ago 324MB
learn/visual_init v1 56a4eab7dc5b 44 hours ago 321MB
registry.cn-beijing.aliyuncs.com/zhangsp/aiwen visual_init 56a4eab7dc5b 44 hours ago 321MB
ubuntu latest 14f60031763d 5 days ago 120MB
hello-world latest 1815c82652c0 5 weeks ago 1.84kB
啟動兩份docker_worker
# docker run -it --name docker_worker1 -h docker_worker1 --net my_network --add-host docker_message:172.18.0.2 learn/worker:v1 /bin/sh /etc/rc.local
# docker run -it --name docker_worker2 -h docker_worker2 --net my_network --add-host docker_message:172.18.0.2 learn/worker:v1 /bin/sh /etc/rc.local
5，執行測試模塊
截止目前，Docker簡單集群已經搭建OK。我們共有三個Docker容器，分別是docker_message、docker_worker1、docker_worker2，如下：
root@apple:/home/kevin/docker/docker_cluster# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b454ac04f6a0 learn/worker:v1 "/bin/sh /etc/rc.l..." About an hour ago Up About an hour 6379/tcp docker_worker2
ecdb7a63da09 learn/worker:v1 "/bin/sh /etc/rc.l..." About an hour ago Up About an hour 6379/tcp docker_worker1
ad7291dea597 learn/redis:v2 "/bin/sh /etc/rc.l..." 3 hours ago Up 3 hours 0.0.0.0:26379->6379/tcp docker_message
cbbbe7a5d47a learn/nginx:v2 "/bin/sh /etc/rc.l..." 7 hours ago Exited (255) 4 hours ago 0.0.0.0:8000->80/tcp, 0.0.0.0:8001->81/tcp nginx_test
93a1b9d39683 ubuntu "bash" 2 days ago Exited (0) 8 hours ago zealous_noether
abdc084f9821 hello-world "/hello" 2 days ago Exited (0) 27 hours ago sleepy_clarke
測試一下Docker集群服務是否正常，可在另外一臺機器上，通過redis-cli測試Docker集群（3個docker容器）。當然可以通過python或其他程序連接redis測試。
kevin@orange:~/docker$ redis-cli -h 192.168.1.89 -p 26379
192.168.1.89:26379> keys *
1) "list_message_out"
2) "list_message_alive"
192.168.1.89:26379> lrange list_message_alive 0 3
1) "docker_worker1 is alive at 2017-07-26 08:34:25"
2) "docker_worker2 is alive at 2017-07-26 08:34:24"
3) "docker_worker1 is alive at 2017-07-26 08:34:24"
4) "docker_worker2 is alive at 2017-07-26 08:34:23"
192.168.1.89:26379> lpush list_message_in "1+2+3"
(integer) 1
192.168.1.89:26379> lrange list_message_out 0 3
1) "worker docker_worker1 return 1+2+3=6"
2) "worker docker_worker1 return 1+2-3+3*3=9"
3) "1+2+3*2=9"
4) "1+2+3*2"
192.168.1.89:26379> lpush list_message_in "1+2+3"
(integer) 1
192.168.1.89:26379> lpush list_message_in "1+2+3"
(integer) 1
192.168.1.89:26379> lpush list_message_in "1+2+3"
(integer) 1
192.168.1.89:26379> lpush list_message_in "1+2+3"
(integer) 1
192.168.1.89:26379> lrange list_message_out 0 3
1) "worker docker_worker1 return 1+2+3=6"
2) "worker docker_worker2 return 1+2+3=6"
3) "worker docker_worker1 return 1+2+3=6"
4) "worker docker_worker2 return 1+2+3=6"
測試的Python代碼如下：
#coding:utf8
import redis, time
def test():
# 連接redis
docker_message = "192.168.1.89"
# docker_message = "docker_message"
r = redis.Redis(host=docker_message, port=26379, db=0)
# list_message_in是docker集群的消息輸入列表
# list_message_out是docker集群的消息輸出列表
# list_message_alive是docker_worker的心跳存活數據列表
list_message_in = "list_message_in"
list_message_out = "list_message_out"
list_message_alive = "list_message_alive"
while(True):
# r.lpush(list_message_in, "1+2+2+10/2")
str_response = r.brpop([list_message_out,list_message_alive])
print time.strftime( "%Y-%m-%d %H:%M:%S", time.localtime(time.time()) ), str_response
time.sleep(0.2)
if __name__ == "__main__":
test()
通過上述測試，可以發現：
- 能夠正常連接到docker_message模塊，通過192.168.1.89:26379，或者172.18.0.1:26378，或者172.18.0.2:6378均可以正常連接。
- 通過list_message_alive列表，可以發現docker_worker1和docker_worker2均正常存活，有心跳報告，并且list_message_alive的列表長度不超過100。
- 通過向list_message_in列表增加數據，可以看到list_message_out有數據返回，說明集群運行正常。
- 通過list_message_out的返回數據，可以看到來自不同的worker，包含docker_worker1和docker_worker2，說明簡單的docker集群已經OK。
6，一些坑
如何run一個鏡像的時候，可以在容器中啟動多個進程
首先，由容器創建鏡像之前，最好修改容器中的/etc/rc.local文件，在exit之前執行/bin/bash，在/bin/bash之前可以啟動自己的服務，例如redis或Python腳本。然后，執行run鏡像的時候，增加/bin/sh /etc/rc.local即可。容器和鏡像的規范化管理
首先，鏡像的命名可以采用learn/redis:v1的方式來命名，learn是項目名稱，redis是模塊名稱，v1是版本號。然后，容器的命名可以采用模塊名稱進行命名。最后，不用的鏡像和容器，盡可能刪除掉，避免越多越亂。
