FastDFS是一個(gè)開源的輕量級分布式文件系統(tǒng)，它對文件進(jìn)行管理，功能包括：文件存儲、文件同步、文件訪問（文件上傳、文件下載）等，解決了大容量存儲和負(fù)載均衡的問題，同時(shí)也能做到在集群環(huán)境下一臺機(jī)子上傳文件，同時(shí)該組下的其他節(jié)點(diǎn)下也備份了上傳的文件。做分布式系統(tǒng)開發(fā)時(shí)，其中要解決的一個(gè)問題就是圖片、音視頻、文件共享的問題和數(shù)據(jù)備份，分布式文件系統(tǒng)正好可以解決這個(gè)需求。FastDFS的服務(wù)主要有兩個(gè)角色Tracker和Storage，Tracker服務(wù)用于負(fù)責(zé)調(diào)度storage節(jié)點(diǎn)與client通信，在訪問上起負(fù)載均衡的作用，和記錄storage節(jié)點(diǎn)的運(yùn)行狀態(tài)，是連接client和storage節(jié)點(diǎn)的樞紐，Storage用于保存文件
-
FastDFS集群部署
-
整體部署模塊圖
FastDFS部署示意圖.png -
環(huán)境準(zhǔn)備
-
名稱 描述 centos系統(tǒng)版本 6.9 libfatscommon FastDFS分離出的一些公用函數(shù)包 FastDFS FastDFS主程序 fastdfs-nginx-module FastDFS和nginx的關(guān)聯(lián)模塊 nginx nginx1.15.5 安裝編譯環(huán)境
yum install git gcc gcc-c++ make automake autoconf libtool pcre pcre-devel zlib zlib-devel openssl-devel wget vim -y-
磁盤安裝路徑說明
說明 位置 FastDFS所以安裝包安裝位置 /usr/local/src tracker數(shù)據(jù) /data/fdfs/tracker Storage數(shù)據(jù) /data/fdfs/Storage 配置文件路徑 /etc/fdfs
-
-
安裝libfatscommon
-
解壓、安裝
unzip libfastcommon-master.zip cd libfastcommon-master ./make.sh && ./make.sh install #編譯安裝
-
安裝FastDFS
-
解壓、安裝
unzip fastdfs-master.zip cd fastdfs-master ./make.sh && ./make.sh install #編譯安裝 cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf #客戶端文件，測試用 cp /usr/local/src/fastdfs/conf/http.conf /etc/fdfs/ #供nginx訪問使用 cp /usr/local/src/fastdfs/conf/mime.types /etc/fdfs/ #供nginx訪問使用
etc目錄下fdfs目錄.png
-
-
安裝fastdfs-nginx-module
-
解壓、安裝
unzip fastdfs-nginx-module-master.zip cp /usr/local/src/fastdfs-nginx-module-master/src/mod_fastdfs.conf /etc/fdfs #復(fù)制配置文件到fdfs目錄
-
安裝nginx
-
解壓、安裝
tar -zxvf nginx-1.15.5.tar.gz cd nginx-1.15.5 #添加fastdfs-nginx-module模塊 ./configure --add-module=/usr/local/src/fastdfs-nginx-module-master/src/ make && make install #編譯安裝
-
FastDFS集群部署配置
- tracker配置
#服務(wù)器ip為 xxx.xxx.78.12, xxx.xxx.78.13 vim /etc/fdfs/tracker.conf #需要修改的內(nèi)容如下 port=22122 # tracker服務(wù)器端口（默認(rèn)22122,一般不修改） base_path=/data/fdfs/tracker #存儲日志和數(shù)據(jù)的根目錄tracker配置.png Storage配置
```
vim /etc/fdfs/storage.conf
#需要修改的內(nèi)容如下
port=23000 # storage服務(wù)端口（默認(rèn)23000,一般不修改）
base_path=/data/fdfs/storage # 數(shù)據(jù)和日志文件存儲根目錄
store_path0=/data/fdfs/storage # 第一個(gè)存儲目錄
tracker_server=xxx.xxx.78.12:22122 # 服務(wù)器1
tracker_server=xxx.xxx.78.13:22122 # 服務(wù)器2
http.server_port=8888 # http訪問文件的端口(默認(rèn)8888,看情況修改,和nginx中保持一致)
```
- client配置
```
vim /etc/fdfs/client.conf
#需要修改的內(nèi)容如下
base_path=/home/moe/dfs
tracker_server=xxx.xxx.78.12:22122 # 服務(wù)器1
tracker_server=xxx.xxx.78.13:22122 # 服務(wù)器2
```
- 配置nginx訪問
```
vim /etc/fdfs/mod_fastdfs.conf
#需要修改的內(nèi)容如下
tracker_server=xxx.xxx.78.12:22122 # 服務(wù)器1
tracker_server=xxx.xxx.78.13:22122 # 服務(wù)器2
url_have_group_name=true
store_path0=/data/fdfs/storage
#配置nginx.config
vim /usr/local/nginx/conf/nginx.conf
#添加如下配置
server {
listen 8888; ## 該端口為storage.conf中的http.server_port相同
server_name localhost;
location ~/group[0-9]/ {
ngx_fastdfs_module;
}
......
......
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
```
-
啟動(dòng)服務(wù)、測試
啟動(dòng)之前我們還需要在防火墻開通端口 vim /etc/sysconfig/iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 22122 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 23000 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 8888 -j ACCEPT service iptables restart #重啟防火墻防火墻端口.png
-
每個(gè)服務(wù)的啟動(dòng)、關(guān)閉和重啟操作
#tracker /etc/init.d/fdfs_trackerd start #啟動(dòng)tracker服務(wù) /etc/init.d/fdfs_trackerd restart #重啟動(dòng)tracker服務(wù) /etc/init.d/fdfs_trackerd stop #停止tracker服務(wù) chkconfig fdfs_trackerd on #自啟動(dòng)tracker服務(wù) #storage /etc/init.d/fdfs_storaged start #啟動(dòng)storage服務(wù) /etc/init.d/fdfs_storaged restart #重動(dòng)storage服務(wù) /etc/init.d/fdfs_storaged stop #停止動(dòng)storage服務(wù) chkconfig fdfs_storaged on #自啟動(dòng)storage服務(wù) #nginx /usr/local/nginx/sbin/nginx #啟動(dòng)nginx /usr/local/nginx/sbin/nginx -s reload #重啟nginx /usr/local/nginx/sbin/nginx -s stop #停止nginx ```  -
檢測集群
# 會顯示會有幾臺storage服務(wù)器,有2臺就會顯示 Storage 1-Storage 2的詳細(xì)信息 /usr/bin/fdfs_monitor /etc/fdfs/storage.conf檢測集群1.png
檢測集群2.png -
圖片上傳測試
#上傳成功返回 文件訪問 ID # fdfs_upload_file 客戶端配置文件 上傳文件路徑 fdfs_upload_file /etc/fdfs/client.conf /data/test.png上傳文件成功.png
- 測試文件訪問
測試nginx默認(rèn)端口80 訪問剛剛上傳的文件，兩個(gè)地址都能訪問通一個(gè)文件，達(dá)到數(shù)據(jù)備份目的。http://xxx.xxx.78.12/group1/M00/00/00/rB9ODFvXuSiAWBYBAALSAkm_6RQ360.png http://xxx.xxx.78.13/group1/M00/00/00/rB9ODFvXuSiAWBYBAALSAkm_6RQ360.png
至此，F(xiàn)astDFS服務(wù)器部署完成
-
FastDFS客戶端集成到SpringBoot
-
首先根據(jù)官方源碼提示，我們先下載源碼使用maven編譯成jar包放到公司maven私服（Nexus），或者你本地的maven私服（也有其他ant等方式，具體請查看github）
FastDFS-java-client-SDK源碼下載地址#編譯jar包（解壓下載的FastDFS-java-client-SDK源碼，使用mvn命令需要先有maven環(huán)境） mvn clean install編譯打包FastDFS-java-client.png
fastdfs-client-java打包成功.png -
maven項(xiàng)目pom.xml中添加依賴
<dependency> <groupId>org.csource</groupId> <artifactId>fastdfs-client-java</artifactId> <version>1.27-SNAPSHOT</version> </dependency> -
接下來我們在項(xiàng)目resources目錄下添加fdfs_client.conf文件
connect_timeout = 30 network_timeout = 30 charset = UTF-8 http.tracker_http_port = 80 http.anti_steal_token = no http.secret_key = 123456 #前面配置的集群tracker服務(wù)器地址 tracker_server = xxx.xxx.78.12:22122 tracker_server = xxx.xxx.78.13:22122 -
寫一個(gè)上傳文件對象類
/** * @Author: maoqitian * @Date: 2018/10/26 0026 17:57 * @Description: FastDFS 文件類 */ public class FastDFSFileEntity { //文件名稱 private String name; //內(nèi)容 private byte[] content; //文件類型 private String ext; //md5值 private String md5; //作者 private String author; public FastDFSFileEntity(String name, byte[] content, String ext, String height, String width, String author) { super(); this.name = name; this.content = content; this.ext = ext; this.author = author; } public FastDFSFileEntity(String name, byte[] content, String ext) { super(); this.name = name; this.content = content; this.ext = ext; } public String getName() { return name; } public void setName(String name) { this.name = name; } public byte[] getContent() { return content; } public void setContent(byte[] content) { this.content = content; } public String getExt() { return ext; } public void setExt(String ext) { this.ext = ext; } public String getMd5() { return md5; } public void setMd5(String md5) { this.md5 = md5; } public String getAuthor() { return author; } public void setAuthor(String author) { this.author = author; } } -
編寫FastDFS操作類，主要是加載初始化配置Tracker服務(wù)器，文件上傳，下載，刪除等操作工具類
/** * @Author: maoqitian * @Date: 2018/10/29 0029 9:30 * @Description: FastDFS 操作類 */ public class FastDFSClient { private static org.slf4j.Logger logger = LoggerFactory.getLogger(FastDFSClient.class); //雙重守護(hù)單例 private static volatile FastDFSClient mInstance; /** * 加載配置信息 **/ static { try { String filePath=new ClassPathResource("fdfs_client.conf").getFile().getAbsolutePath(); ClientGlobal.init(filePath); }catch (Exception e){ logger.error("FastDFS Client Init Fail!",e); } } private FastDFSClient(){ } public static FastDFSClient getInstance(){ if(mInstance == null){ synchronized (FastDFSClient.class){ if(mInstance == null){ mInstance=new FastDFSClient(); } } } return mInstance; } /** * @Author maoqitian * @Description 上傳文件 * @Date 2018/10/29 0029 9:42 * @Param [fastDFSFileEntity] * @return java.lang.String[] **/ public String[] upload(FastDFSFileEntity file){ logger.info("File Name: " + file.getName() + "File Length:" + file.getContent().length); NameValuePair[] metalist=new NameValuePair[1]; metalist[0]=new NameValuePair("author",file.getAuthor()); long startTime = System.currentTimeMillis(); String[] uploadResults= null; StorageClient storageClient=null; try { storageClient=getTrackerClient(); uploadResults = storageClient.upload_file(file.getContent(),file.getExt(),metalist); }catch (IOException e){ logger.error("IO Exception when uploadind the file:"+file.getName(),e); } catch (Exception e){ logger.error("Non IO Exception when uploadind the file:"+file.getName(),e); } logger.info("upload_file time used:" + (System.currentTimeMillis() - startTime) + " ms"); if(uploadResults==null && storageClient!=null){ logger.error("upload file fail, error code:" + storageClient.getErrorCode()); } String groupName = uploadResults[0]; String remoteFileName = uploadResults[1]; logger.info("upload file successfully!!!" + "group_name:" + groupName + ", remoteFileName:" + " " + remoteFileName); return uploadResults; } public FileInfo getFile(String groupName, String remoteFileName) { try { StorageClient storageClient = getTrackerClient(); return storageClient.get_file_info(groupName, remoteFileName); } catch (IOException e) { logger.error("IO Exception: Get File from Fast DFS failed", e); } catch (Exception e) { logger.error("Non IO Exception: Get File from Fast DFS failed", e); } return null; } public InputStream downFile(String groupName, String remoteFileName) { try { StorageClient storageClient = getTrackerClient(); byte[] fileByte = storageClient.download_file(groupName, remoteFileName); InputStream ins = new ByteArrayInputStream(fileByte); return ins; } catch (IOException e) { logger.error("IO Exception: Get File from Fast DFS failed", e); } catch (Exception e) { logger.error("Non IO Exception: Get File from Fast DFS failed", e); } return null; } /** * @Author maoqitian * @Description * @Date 2018/10/31 0031 11:19 * @Param [remoteFileName] * @return int -1 失敗 0成功 **/ public int deleteFile(String remoteFileName) throws Exception { StorageClient storageClient = getTrackerClient(); int i = storageClient.delete_file("group1", remoteFileName); logger.info("delete file successfully!!!" + i); return i; } public StorageServer[] getStoreStorages(String groupName) throws IOException { TrackerClient trackerClient = new TrackerClient(); TrackerServer trackerServer = trackerClient.getConnection(); return trackerClient.getStoreStorages(trackerServer, groupName); } public ServerInfo[] getFetchStorages(String groupName, String remoteFileName) throws IOException { TrackerClient trackerClient = new TrackerClient(); TrackerServer trackerServer = trackerClient.getConnection(); return trackerClient.getFetchStorages(trackerServer, groupName, remoteFileName); } public String getTrackerUrl() throws IOException { return "http://"+getTrackerServer().getInetSocketAddress().getHostString()+":"+ClientGlobal.getG_tracker_http_port()+"/"; } /** * @Author maoqitian * @Description 獲取 StorageClient * @Date 2018/10/29 0029 10:33 * @Param [] * @return org.csource.fastdfs.StorageClient **/ private StorageClient getTrackerClient() throws IOException{ TrackerServer trackerServer=getTrackerServer(); StorageClient storageClient=new StorageClient(trackerServer,null); return storageClient; } /** * @Author maoqitian * @Description 獲取 TrackerServer * @Date 2018/10/29 0029 10:34 * @Param [] * @return org.csource.fastdfs.TrackerServer **/ private TrackerServer getTrackerServer() throws IOException { TrackerClient trackerClient=new TrackerClient(); TrackerServer trackerServer = trackerClient.getConnection(); return trackerServer; } -
Controller編寫，接收請求并上傳文件返回文件訪問路徑（這里寫一個(gè)文件上傳的例子，其他文件下載，刪除等功能可根據(jù)自己需求進(jìn)行編寫）
/** * @Author maoqitian * @Description 上傳文件 * @Date 2018/10/30 0030 15:07 * @Param [file] * @return com.gxxmt.common.utils.ResultApi **/ @RequestMapping("/upload") public ResultApi upload(@RequestParam("file") MultipartFile file) throws Exception { if (file.isEmpty()) { throw new RRException("上傳文件不能為空"); } String url; //此處域名獲取可以根據(jù)自需求編寫 String domainUrl = OSSFactory.build().getDomainPath(); logger.info("配置的域名為"+domainUrl); if (StringUtils.isNotBlank(domainUrl)){ url = uploadFile(file,domainUrl); return ResultApi.success.put("url",url); }else { return ResultApi.error("域名配置為空,請先配置對象存儲域名"); } } /** * @Author maoqitian * @Description 上傳文件到 FastDFS * @Date 2018/10/29 0029 11:11 * @Param [file] * @Param [domainName] 域名 * @return path 文件訪問路徑 **/ public String uploadFile(MultipartFile file,String domainName) throws IOException { String[] fileAbsolutePath={}; String fileName=file.getOriginalFilename(); String ext=fileName.substring(fileName.lastIndexOf(".")+1); byte[] file_buff=null; InputStream inputStream = file.getInputStream(); if(inputStream!=null){ int available = inputStream.available(); file_buff=new byte[available]; inputStream.read(file_buff); } inputStream.close(); FastDFSFileEntity fastDFSFileEntity=new FastDFSFileEntity(fileName,file_buff,ext); try { fileAbsolutePath=FastDFSClient.getInstance().upload(fastDFSFileEntity); logger.info(fileAbsolutePath.toString()); }catch (Exception e){ logger.error("upload file Exception!",e); throw new RRException("文件上傳出錯(cuò)"+e); } if(fileAbsolutePath == null){ logger.error("upload file failed,please upload again!"); throw new RRException("文件上傳失敗，請重新上傳"); } String path=domainName+fileAbsolutePath[0]+ "/"+fileAbsolutePath[1]; return path; }
-
-
測試一下該方法，上傳一個(gè)圖片
-
由日志打印我們可以看出圖片已經(jīng)上傳成功
FastDFS-java-client 上傳圖片成功.png -
測試訪問上傳的圖片
測試上傳的圖片是否可以進(jìn)行訪問.png
-
到此，F(xiàn)astDFS服務(wù)器集群部署和集成客戶端到SpringBoot中已經(jīng)完成，以后我們就可以愉快的使用FastDFS服務(wù)保存我們的圖片等并備份。如果文章中有寫得不對的地方，請給我留言指出，大家一起學(xué)習(xí)進(jìn)步。如果覺得我的文章給予你幫助，也請給我一個(gè)喜歡和關(guān)注。
