嗯灬啊灬把腿张开灬a片梅花三弄,娇妻被粗大爽ⅹxoo网站,少妇裸交aa大片

1.搭建環境：

系統版本：Linux centos-linux.shared 3.10.0-123.el7.x86_64 #1 SMP Mon Jun 30 12:09:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

python版本：Python 3.5.1

1.1.搭建python3環境：

本人在嘗試過后選擇集成環境Anaconda

1.1.1.編譯

# 下載依賴
yum install -y ncurses-devel openssl openssl-devel zlib-devel gcc make glibc-devel libffi-devel glibc-static glibc-utils sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-deve
# 下載python版本
wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz
# 或者使用國內源
wget http://mirrors.sohu.com/python/3.5.1/Python-3.5.1.tgz
mv Python-3.5.1.tgz /usr/local/src;cd /usr/local/src
# 解壓
tar -zxf Python-3.5.1.tgz;cd Python-3.5.1
# 編譯安裝
./configure --prefix=/usr/local/python3.5 --enable-shared
make && make install
# 建立軟鏈接
ln -s /usr/local/python3.5/bin/python3 /usr/bin/python3
echo "/usr/local/python3.5/lib" > /etc/ld.so.conf.d/python3.5.conf
ldconfig
# 驗證python3
python3
# Python 3.5.1 (default, Oct  9 2016, 11:44:24)
# [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux
# Type "help", "copyright", "credits" or "license" for more information.
# >>>
# pip
/usr/local/python3.5/bin/pip3 install --upgrade pip
ln -s /usr/local/python3.5/bin/pip /usr/bin/pip
# 本人在安裝時出現問題 將pip重裝
wget https://bootstrap.pypa.io/get-pip.py --no-check-certificate
python get-pip.py

1.1.2.集成環境anaconda

# 集成環境anaconda(推薦)
wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
# 直接安裝即可
./Anaconda3-4.2.0-Linux-x86_64.sh
# 若出錯，可能是解壓失敗
yum install bzip2

1.2.安裝mariaDB

# 安裝
yum -y install mariadb mariadb-server
# 啟動
systemctl start mariadb
# 設置為開機啟動
systemctl enable mariadb
# 配置密碼 默認為空
mysql_secure_installation
# 登錄
mysql -u root -p
# 創建一個用戶 自己設定賬戶密碼
CREATE USER 'user_name'@'localhost' IDENTIFIED BY 'user_pass';
GRANT ALL PRIVILEGES ON *.* TO 'user_name'@'localhost' WITH GRANT OPTION;
CREATE USER 'user_name'@'%' IDENTIFIED BY 'user_pass';
GRANT ALL PRIVILEGES ON *.* TO 'user_name'@'%' WITH GRANT OPTION;

1.3.安裝pyspider

本人使用Anaconda

# 搭建虛擬環境sbird python版本3.*
conda create -n sbird python=3*
# 進入環境
source activate sbird
# 安裝pyspider
pip install pyspider
# 報錯 
# it does not exist.  The exported locale is "en_US.UTF-8" but it is not supported
# 執行 可寫入.bashrc
export LC_ALL=en_US.utf-8
export LANG=en_US.utf-8
#ImportError: pycurl: libcurl link-time version (7.29.0) is older than compile-time version (7.49.0)
conda install pycurl
# 退出
source deactivate sbird
# 若在虛擬機內 出現無法訪問localhost:5000 可關閉防火墻
systemctl stop firewalld.service
#########直接運行源碼==============
mkdir git;cd git
# 下載
git clone https://github.com/binux/pyspider.git
# 安裝
/root/anaconda3/envs/sbird/bin/python  /root/git/pyspider/run.py

其他方法

# 搭建虛擬環境
pip install virtualenv
mkdir python;cd python
# 創建虛擬環境pyenv3
virtualenv -p /usr/bin/python3 pyenv3
# 進入虛擬環境 激活環境
cd pyenv3/
source ./bin/activate
pip install pyspider
# 若pycurl報錯 
yum install libcurl-devel
# 繼續
pip install pyspider
# 關閉
deactivate

本人推薦用anaconda方式安裝

若pyspider運行過程中出現錯誤，參考anaconda安裝部分，至此，訪問localhost:5000可看到頁面。

1.4.安裝Supervisor

# 安裝
yum install supervisor -y
# 若無法檢索 則添加阿里的epel源
vim /etc/yum.repos.d/epel.repo
# 添加以下內容
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=http://mirrors.aliyun.com/epel/7/$basearch
http://mirrors.aliyuncs.com/epel/7/$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug
http://mirrors.aliyuncs.com/epel/7/$basearch/debug
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0

[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=http://mirrors.aliyun.com/epel/7/SRPMS
http://mirrors.aliyuncs.com/epel/7/SRPMS
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
# 安裝
yum install supervisor -y
# 測試是否安裝成功
echo_supervisord_conf

1.4.1.Supervisor用法

supervisord     #supervisor的服務器端部分 啟動
supervisorctl   #啟動supervisor的命令行窗口
# 假設創建進程pyspider01
vim /etc/supervisord.d/pyspider01.ini
# 寫入以下內容
[program:pyspider01]

command      = /root/anaconda3/envs/sbird/bin/python  /root/git/pyspider/run.py
directory    = /root/git/pyspider
user         = root
process_name = %(program_name)s
autostart    = true
autorestart  = true
startsecs    = 3

redirect_stderr         = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups  = 10
stdout_logfile          = /pyspider/supervisor/pyspider01.log
# 重載
supervisorctl reload
# 啟動
supervisorctl start pyspider01
# 也可這樣啟動
supervisord -c /etc/supervisord.conf
# 查看狀態
supervisorctl status
# output 
pyspider01                       RUNNING   pid 4026, uptime 0:02:40
# 關閉
supervisorctl shutdown

1.5.安裝redis

# 消息隊列采用redis
mkdir download;cd download
wget http://download.redis.io/releases/redis-3.2.4.tar.gz
tar xzf redis-3.2.4.tar.gz
cd redis-3.2.4
make
# 或者直接yum安裝
yum -y install redis
# 啟動
systemctl start redis.service
# 重啟
systemctl restart redis.service
# 停止
systemctl stop redis.service
# 查看狀態
systemctl status redis.service
# 更改文件/etc/redis.conf
vim /etc/redis.conf
# 更改內容
daemonize no 改為 daemonize yes
bind 127.0.0.1 改為 bind 10.211.55.22(當前服務器ip)
# 重啟redis
systemctl restart redis.service

1.6.關于自啟動

# Supervisor添加到自啟動服務
systemctl enable supervisord.service
# redis添加到自啟動服務
systemctl enable redis.service
# 關閉防火墻自啟動
systemctl disable firewalld.service

至此，pyspider單個服務器運行環境搭建且部署完畢，啟動localhost:5000進入web界面。

也可編寫腳本運行，在/pyspider/supervisor/pyspider01.log查看運行狀態。

2.分布式部署

剛才配置的服務器，將其命名為centos01，按照這樣的配置，再分別部署兩臺centos02、centos03。

如下：

服務器名稱	ip	說明
centos01	10.211.55.22	redis,mariaDB, scheduler
centos02	10.211.55.23	fetcher, processor, result_worker,phantomjs
centos03	10.211.55.24	fetcher, processor,,result_worker,webui

2.1.centos01

進入服務器centos01，經過第一步，基本環境已經搭好，首先編輯配置文件/pyspider/config.json

{
  "taskdb": "mysql+taskdb://user_name:user_pass@10.211.55.22:3306/taskdb",
  "projectdb": "mysql+projectdb://user_name:user_pass@10.211.55.22:3306/projectdb",
  "resultdb": "mysql+resultdb://user_name:user_pass@10.211.55.22:3306/resultdb",
  "message_queue": "redis://10.211.55.22:6379/db",
  "logging-config": "/pyspider/logging.conf",
  "phantomjs-proxy":"10.211.55.23:25555",
  "webui": {
    "username": "",
    "password": "",
    "need-auth": false,
    "host":"10.211.55.24",
    "port":"5000",
    "scheduler-rpc":"http:// 10.211.55.22:5002",
    "fetcher-rpc":"http://10.211.55.23:5001"
  },
  "fetcher": {
    "xmlrpc":true,
    "xmlrpc-host": "0.0.0.0",
    "xmlrpc-port": "5001"
  },
  "scheduler": {
    "xmlrpc":true,
    "xmlrpc-host": "0.0.0.0",
    "xmlrpc-port": "5002"
  }
}

嘗試運行下：

/root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json scheduler
# 報錯
ImportError: No module named 'mysql'
# 下載 mysql-connector-python
cd ~/git/
git clone https://github.com/mysql/mysql-connector-python.git
# 安裝
source activate sbird
cd mysql-connector-python
python setup.py install
# 安裝redis
pip install redis
source deactivate
# 運行
/root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json scheduler
# 輸出 ok
[I 161010 15:57:25 scheduler:644] scheduler starting...
[I 161010 15:57:25 scheduler:779] scheduler.xmlrpc listening on 0.0.0.0:5002
[I 161010 15:57:25 scheduler:583] in 5m: new:0,success:0,retry:0,failed:0

運行成功后，可直接更改/etc/supervisord.d/pyspider01.ini如下：

[program:pyspider01]

command      = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json scheduler
directory    = /root/git/pyspider
user         = root
process_name = %(program_name)s
autostart    = true
autorestart  = true
startsecs    = 3

redirect_stderr         = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups  = 10
stdout_logfile          = /pyspider/supervisor/pyspider01.log

# 重載
supervisorctl reload
# 查看狀態
supervisorctl status

centos01部署完畢。

2.2.centos02

在centos02中，需要運行result_worker、processor、phantomjs、fetcher

分別建立文件：

/etc/supervisord.d/result_worker.ini

[program:result_worker]

command      = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json result_worker
directory    = /root/git/pyspider
user         = root
process_name = %(program_name)s
autostart    = true
autorestart  = true
startsecs    = 3

redirect_stderr         = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups  = 10
stdout_logfile          = /pyspider/supervisor/result_worker.log

/etc/supervisord.d/processor.ini

[program:processor]

command      = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json processor
directory    = /root/git/pyspider
user         = root
process_name = %(program_name)s
autostart    = true
autorestart  = true
startsecs    = 3

redirect_stderr         = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups  = 10
stdout_logfile          = /pyspider/supervisor/processor.log

/etc/supervisord.d/phantomjs.ini

[program:phantomjs]

command      = /pyspider/phantomjs --config=/pyspider/pjsconfig.json /pyspider/phantomjs_fetcher.js 25555
directory    = /root/git/pyspider
user         = root
process_name = %(program_name)s
autostart    = true
autorestart  = true
startsecs    = 3

redirect_stderr         = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups  = 10
stdout_logfile          = /pyspider/supervisor/phantomjs.log

/etc/supervisord.d/fetcher.ini

[program:fetcher]

command      = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json fetcher
directory    = /root/git/pyspider
user         = root
process_name = %(program_name)s
autostart    = true
autorestart  = true
startsecs    = 3

redirect_stderr         = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups  = 10
stdout_logfile          = /pyspider/supervisor/fetcher.log

在pyspider目錄中建立pjsconfig.json

{
  /*--ignore-ssl-errors=true */
  "ignoreSslErrors": true,
  
  /*--ssl-protocol=true */
  "sslprotocol": "any",
  
  /* Same as: --output-encoding=utf8 */
  "outputEncoding": "utf8",
  
  /* persistent Cookies. */
  /*cookiesfile="e:/phontjscookies.txt",*/
  cookiesfile="pyspider/phontjscookies.txt",
  
  /* load image */
  autoLoadImages = false
}

下載phantomjs至/pyspider/文件夾，將git/pyspider/pyspider/fetcher/phantomjs_fetcher.js復制到phantomjs_fetcher.js

# 重載
supervisorctl reload
# 查看狀態
supervisorctl status
# output
fetcher                          RUNNING   pid 3446, uptime 0:00:07
phantomjs                        RUNNING   pid 3448, uptime 0:00:07
processor                        RUNNING   pid 3447, uptime 0:00:07
result_worker                    RUNNING   pid 3445, uptime 0:00:07

centos02部署完畢。

2.3.centos03

部署這三個進程fetcher, processor, result_worker和centos02 一樣，本服務器主要是在前面的基礎上加上webui

建立文件：

/etc/supervisord.d/webui.ini

[program:webui]

command      = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json webui
directory    = /root/git/pyspider
user         = root
process_name = %(program_name)s
autostart    = true
autorestart  = true
startsecs    = 3

redirect_stderr         = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups  = 10
stdout_logfile          = /pyspider/supervisor/webui.log

# 重載
supervisorctl reload
# 查看狀態
supervisorctl status
# output
fetcher                          RUNNING   pid 2724, uptime 0:00:07
processor                        RUNNING   pid 2725, uptime 0:00:07
result_worker                    RUNNING   pid 2723, uptime 0:00:07
webui                            RUNNING   pid 2726, uptime 0:00:07

3.總結

訪問 http://10.211.55.24:5000 即可，盡情爬取吧。

pyspider

三个男躁一个女,国精产品一区一手机的秘密,麦子交换系列最经典十句话,欧美国产综合欧美视频

centos7分布式部署pyspider

centos7分布式部署pyspider

1.搭建環境：

1.1.搭建python3環境：

1.1.1.編譯

1.1.2.集成環境anaconda

1.2.安裝mariaDB

1.3.安裝pyspider

1.4.安裝Supervisor

1.4.1.Supervisor用法

1.5.安裝redis

1.6.關于自啟動

2.分布式部署

2.1.centos01

2.2.centos02

2.3.centos03

3.總結

推薦閱讀更多精彩內容

三个男躁一个女,国精产品一区一手机的秘密,麦子交换系列最经典十句话,欧美 国产 综合 欧美 视频

centos7分布式部署pyspider

1.搭建環境：

1.1.搭建python3環境：

1.1.1.編譯

1.1.2.集成環境anaconda

1.2.安裝mariaDB

1.3.安裝pyspider

1.4.安裝Supervisor

1.4.1.Supervisor用法

1.5.安裝redis

1.6.關于自啟動

2.分布式部署

2.1.centos01

2.2.centos02

2.3.centos03

3.總結

推薦閱讀更多精彩內容

三个男躁一个女,国精产品一区一手机的秘密,麦子交换系列最经典十句话,欧美国产综合欧美视频