python爬蟲之路--準備環境

安裝Python3

1.MAC下安裝python3

MAC下推薦使用homebrew來安裝Python3,什么是homebrew呢?
它是macOS 缺失的軟件包管理器。如何安裝homebrew,只需要在終端輸入:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

如果想進一步了解homebrew,點擊 homebrew官網

安裝完畢homebrew之后,開始安裝python3

1.1搜索包:
brew searsh python3

結果如下,說明存在python3的包,可以安裝。如圖 1-1


圖1-1
1.2安裝包:
brew install python3

控制臺輸入安裝過程如下:

==> Installing dependencies for python: gdbm, openssl, readline, sqlite, xz
==> Installing python dependency: gdbm
==> Downloading https://homebrew.bintray.com/bottles/gdbm-1.14.1_1.high_sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring gdbm-1.14.1_1.high_sierra.bottle.tar.gz
?? /usr/local/Cellar/gdbm/1.14.1_1: 20 files, 555.7KB
==> Installing python dependency: openssl
==> Downloading https://homebrew.bintray.com/bottles/openssl-1.0.2o_1.high_sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring openssl-1.0.2o_1.high_sierra.bottle.tar.gz
==> Caveats
A CA file has been bootstrapped using certificates from the SystemRoots
keychain. To add additional certificates (e.g. the certificates added in
the System keychain), place .pem files in
/usr/local/etc/openssl/certs

and run
/usr/local/opt/openssl/bin/c_rehash

This formula is keg-only, which means it was not symlinked into /usr/local,
because Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries.

If you need to have this software first in your PATH run:
echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.bash_profile

For compilers to find this software you may need to set:
LDFLAGS: -L/usr/local/opt/openssl/lib
CPPFLAGS: -I/usr/local/opt/openssl/include

==> Summary
?? /usr/local/Cellar/openssl/1.0.2o_1: 1,791 files, 12.3MB
==> Installing python dependency: readline
==> Downloading https://homebrew.bintray.com/bottles/readline-7.0.3_1.high_sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring readline-7.0.3_1.high_sierra.bottle.tar.gz
==> Caveats
This formula is keg-only, which means it was not symlinked into /usr/local,
because macOS provides the BSD libedit library, which shadows libreadline.
In order to prevent conflicts when programs look for libreadline we are
defaulting this GNU Readline installation to keg-only.

For compilers to find this software you may need to set:
LDFLAGS: -L/usr/local/opt/readline/lib
CPPFLAGS: -I/usr/local/opt/readline/include

==> Summary
?? /usr/local/Cellar/readline/7.0.3_1: 46 files, 1.5MB
==> Installing python dependency: sqlite
==> Downloading https://homebrew.bintray.com/bottles/sqlite-3.23.1.high_sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring sqlite-3.23.1.high_sierra.bottle.tar.gz
==> Caveats
This formula is keg-only, which means it was not symlinked into /usr/local,
because macOS provides an older sqlite3.

If you need to have this software first in your PATH run:
echo 'export PATH="/usr/local/opt/sqlite/bin:$PATH"' >> ~/.bash_profile

For compilers to find this software you may need to set:
LDFLAGS: -L/usr/local/opt/sqlite/lib
CPPFLAGS: -I/usr/local/opt/sqlite/include

==> Summary
?? /usr/local/Cellar/sqlite/3.23.1: 11 files, 3MB
==> Installing python dependency: xz
==> Downloading https://homebrew.bintray.com/bottles/xz-5.2.4.high_sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring xz-5.2.4.high_sierra.bottle.tar.gz
?? /usr/local/Cellar/xz/5.2.4: 92 files, 1MB
==> Installing python
==> Downloading https://homebrew.bintray.com/bottles/python-3.6.5.high_sierra.bottle.1.tar.gz
######################################################################## 100.0%
==> Pouring python-3.6.5.high_sierra.bottle.1.tar.gz
==> /usr/local/Cellar/python/3.6.5/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/C
==> /usr/local/Cellar/python/3.6.5/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/C
==> /usr/local/Cellar/python/3.6.5/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/C
==> Caveats
Python has been installed as
/usr/local/bin/python3

Unversioned symlinks python, python-config, pip etc. pointing to
python3, python3-config, pip3 etc., respectively, have been installed into
/usr/local/opt/python/libexec/bin

If you need Homebrew's Python 2.7 run
brew install python@2

Pip, setuptools, and wheel have been installed. To update them run
pip3 install --upgrade pip setuptools wheel

You can install Python packages with
pip3 install <package>
They will install into the site-package directory
/usr/local/lib/python3.6/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Summary
?? /usr/local/Cellar/python/3.6.5: 4,736 files, 99.2MB

1.3 檢測安裝是否成功。

打開終端,在命令行界面輸入分別python3 和pip -V 查看,如圖 1-2


圖1-2

安裝請求庫

爬蟲可以大致分為三個步驟:抓取頁面,分析頁面和存儲數據。
在抓取頁面的時候,我們需要模擬瀏覽器向服務器發送請求,這時需要用到一些python的庫來完成這些請求。常見的有:requests, Selenium和aiohttp等

安裝requests

requests是第三方庫,python默認不自帶這個庫,所以需要我們手動安裝這個模塊。相關參考資料1.Github | 2.PyPI | 3.官網文檔 | 4.中文文檔

使用pip安裝requets,執行:

pip3 install requests

驗證安裝,打開終端,在命令行中輸入:python3,進入命令行模式。

>>> import requests

如果什么錯誤提示有沒有,則證明安裝成功的安裝了requests。

安裝Selenium

Selenuim是一個自動化測試工具,利用它我們可以干什么呢?可以驅動瀏覽器執行特定的動作,例如點擊下拉等操作。對于一些Javascript渲染的頁面來說,這種方式很有效。相關參考資料1.官方網站 | 2.Github | 3.PyPI | 4.中文文檔

使用pip安裝selenium,執行

pip3 install selenium

此時報錯了!!!錯誤信息

Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/selenium'
Consider using the `--user` option or check the permissions.

先把這個問題留在這里。

驗證安裝,打開終端,在命令行中輸入:python3,進入命令行模式。

>>> import selenium

沒有報錯,安裝成功!

最后編輯于
?著作權歸作者所有,轉載或內容合作請聯系作者
平臺聲明:文章內容(如有圖片或視頻亦包括在內)由作者上傳并發布,文章內容僅代表作者本人觀點,簡書系信息發布平臺,僅提供信息存儲服務。

推薦閱讀更多精彩內容