當前數據量激增,數據落地的載體是文件,各種終端不斷產生著大量的各種類型各種大小的文件,從幾k的文本或者網頁文件到幾M的圖片文件,幾十M的視頻文件到幾G的高清視頻文件,鏡像文件,不一而足,硬件性價比持續增高,帶寬性價比持續增高的情況下,對文件的高并發低延遲的請求是一種必然的趨勢,而傳統的文件系統難以達到這種要求,分布式文件系統,在云計算時代,要擔當文件系統的使命。
swift
是由大名鼎鼎的Rackspace
在2010年貢獻給openstack
來作為對象存儲系統的,主要優點有:
- 大公司領銜開源,
openstack
核心組件,新浪和韓國電信已經基于swift
做出商用的對象存儲系統。 - 使用一致性hash,選取存取節點,去中心化,方便于水平擴展。
- 有完善的權限控制,支持對接外部權限控制中間件(方便對接本公司的權限控制或認證服務)
-
hadoop
默認支持兼容swift
對象存儲系統(另外還兼容的是Amazon S3
及Azure Blob Storage
),便于后續存儲文件的大數據處理,彌補了hdfs
對大量小文件的存取的低效以及空間的浪費
今天就想介紹下swift
的分布式部署,后續會持續介紹swift的組件架構分析及性能測試等
操作系統:Ubuntu 14.04.4 LTS
安裝拓撲
proxy-swift 10.0.41.20
swift-store-node 10.0.41.21 掛載 sdb sdc 兩個磁盤
swift-store-node 10.0.41.22 掛載 sdb sdc 兩個磁盤
安裝依賴(每個節點都要安裝)
apt-get install software-properties-common
add-apt-repository cloud-archive:mitaka
apt-get update && apt-get dist-upgrade
#安裝apt-get install python-openstackclient(非必需)
apt-get install python-openstackclient
swift-proxy 安裝
apt-get install swift swift-proxy python-swiftclient \
memcached
mkdir /etc/swift
curl -o /etc/swift/proxy-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/proxy-server.conf-sample?h=stable/mitaka
修改配置
vi /etc/swift/proxy-server.conf
修改[DEFAULT]
區域
[DEFAULT]
...
bind_port = 8080
user = swift
swift_dir = /etc/swift
修改[app:proxy-server]
區域
[app:proxy-server]
use = egg:swift#proxy
...
account_autocreate = True
swift-store-node 安裝
格式化,并掛載磁盤
mkfs.xfs -f /dev/sdb
mkfs.xfs -f /dev/sdc
mkdir -p /srv/node/sdb
mkdir -p /srv/node/sdc
vi /etc/fstab
/dev/sdb /srv/node/sdb xfs noatime,nodiratime,nobarrier,logbufs=8 0 2
/dev/sdc /srv/node/sdc xfs noatime,nodiratime,nobarrier,logbufs=8 0 2
mount /srv/node/sdb
mount /srv/node/sdc
修改rsyncd
配置
創建/etc/rsyncd.conf
添加內容
uid = root
gid = root
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = MANAGEMENT_INTERFACE_IP_ADDRESS #(10.0.41.21/22)
[account]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/account.lock
[container]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/container.lock
[object]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/object.lock
vi /etc/default/rsync
#修改 RSYNC_ENABLE=true
service rsync start
安裝依賴
apt-get install swift swift-account swift-container swift-object
curl -o /etc/swift/account-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/account-server.conf-sample?h=stable/mitaka
curl -o /etc/swift/container-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/container-server.conf-sample?h=stable/mitaka
curl -o /etc/swift/object-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/object-server.conf-sample?h=stable/mitaka
修改 /etc/swift/account-server.conf
配置
修改[DEFAULT]
區域
[DEFAULT]
...
bind_ip = MANAGEMENT_INTERFACE_IP_ADDRESS
bind_port = 6002
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True
修改[filter:recon]
區域
[filter:recon]
use = egg:swift#recon
...
recon_cache_path = /var/cache/swift
```
######修改 `/etc/swift/container-server.conf`配置
修改`[DEFAULT]`區域
```shell
[DEFAULT]
...
bind_ip = MANAGEMENT_INTERFACE_IP_ADDRESS
bind_port = 6001
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True
```
修改`[filter:recon]`區域
```shell
[filter:recon]
use = egg:swift#recon
...
recon_cache_path = /var/cache/swift
```
######修改`/etc/swift/object-server.conf`文件
修改`[DEFAULT]`區域
```shell
[DEFAULT]
...
bind_ip = MANAGEMENT_INTERFACE_IP_ADDRESS
bind_port = 6000
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True
```
修改`[filter:recon]`區域
```shell
[filter:recon]
use = egg:swift#recon
...
recon_cache_path = /var/cache/swift
recon_lock_path = /var/lock
```
######創建`cache`目錄
```shell
mkdir -p /var/cache/swift
chmod -R 775 /var/cache/swift
Create and distribute initial rings (在 swiftk-proxy上操作)
cd /etc/swift/
swift-ring-builder account.builder create 10 3 1
swift-ring-builder account.builder add \
--region 1 --zone 1 --ip 10.0.41.21 --port 6002 --device sdb --weight 100
swift-ring-builder account.builder add \
--region 1 --zone 1 --ip 10.0.41.21 --port 6002 --device sdc --weight 100
swift-ring-builder account.builder add \
--region 1 --zone 2 --ip 10.0.41.22 --port 6002 --device sdb --weight 100
swift-ring-builder account.builder add \
--region 1 --zone 2 --ip 10.0.41.22 --port 6002 --device sdc --weight 100
swift-ring-builder account.builder
swift-ring-builder account.builder rebalance
swift-ring-builder container.builder create 10 3 1
swift-ring-builder container.builder add \
--region 1 --zone 1 --ip 10.0.41.21 --port 6001 --device sdb --weight 100
swift-ring-builder container.builder add \
--region 1 --zone 1 --ip 10.0.41.21 --port 6001 --device sdc --weight 100
swift-ring-builder container.builder add \
--region 1 --zone 2 --ip 10.0.41.22 --port 6001 --device sdb --weight 100
swift-ring-builder container.builder add \
--region 1 --zone 2 --ip 10.0.41.22 --port 6001 --device sdc --weight 100
swift-ring-builder container.builder
swift-ring-builder container.builder rebalance
swift-ring-builder object.builder create 10 3 1
swift-ring-builder object.builder add \
--region 1 --zone 1 --ip 10.0.41.21 --port 6000 --device sdb --weight 100
swift-ring-builder object.builder add \
--region 1 --zone 1 --ip 10.0.41.21 --port 6000 --device sdc --weight 100
swift-ring-builder object.builder add \
--region 1 --zone 2 --ip 10.0.41.22 --port 6000 --device sdb --weight 100
swift-ring-builder object.builder add \
--region 1 --zone 2 --ip 10.0.41.22 --port 6000 --device sdc --weight 100
swift-ring-builder object.builder
swift-ring-builder object.builder rebalance
把/etc/swift/*.ring.gz 文件拷貝到 store node 的/etc/swift/ 目錄
cd /etc/swift ; scp root@10.0.41.20:/etc/swift/*.ring.gz ./
finally install (在swift-proxy節點上執行)
curl -o /etc/swift/swift.conf \
https://git.openstack.org/cgit/openstack/swift/plain/etc/swift.conf-sample?h=stable/mitaka
修改/etc/swift/swift.conf
配置
vi /etc/swift/swift.conf
修改[swift-hash]
區域
[swift-hash]
...
swift_hash_path_suffix = HASH_PATH_SUFFIX
swift_hash_path_prefix = HASH_PATH_PREFIX
可用下面的方式生成隨機字符串,替換上面的HASH_PATH_SUFFIX
openssl rand -hex 10
把swift.conf
文件拷貝到所有的store node
上
cd /etc/swift ; scp root@10.0.41.20:/etc/swift/swift.conf ./
啟動swift
- 啟動
swift-proxy
service memcached restart service swift-proxy restart
- 啟動
swift-store-node
swift-init all start