知識(shí)點(diǎn)：lxml，Xpath語法，bsp，threading
爬取 斗圖網(wǎng)的圖片 https://www.doutula.com/article/list?page=1

# coding:utf-8
import requests
import threading # 多線程處理與控制
from lxml import etree # 解析網(wǎng)頁
from bs4 import BeautifulSoup #

# 打開網(wǎng)頁 獲取源碼
def get_html(url):
   # url='https://www.doutula.com/article/list?page=2'
    headers = {'user-agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'}
    request = requests.get(url=url,headers=headers)
    response = request.content
    # print response
    return response

# 從主頁獲取斗圖的每一個(gè)url 獲取源碼
def get_img_html(html):
    soup = BeautifulSoup(html,'lxml') # c創(chuàng)建一個(gè)對(duì)象
    all_a=soup.find_all('a',class_="list-group-item") # 過濾 下劃線 class_ 區(qū)分關(guān)鍵字
    for i in all_a:
        img_html = get_html(i['href'])#找到超鏈接
        # print img_html
        get_img(img_html)

# 獲取每個(gè)圖片
def get_img(html):
    soup = etree.HTML(html) # 初始化打印源碼，自動(dòng)補(bǔ)全
    items = soup.xpath('//div[@class="artile_des"]') # 解析網(wǎng)頁方法 // 選擇 ， 【】 條件， @
    for item in items:
        imgurl = item.xpath('table/tbody/tr/td/a/img/@onerror')
        start_save_img(imgurl)

# 下載圖片
def save_img(img_url):
    img_url = img_url.split('=')[-1][1:-2].replace('jp','jpg')
    print u'正在下載'+'http:'+img_url
    img_content = requests.get('http:'+img_url).content
    with open('doutu/%s.jpg'% img_url.split('/')[-1],'wb') as f:
        f.write(img_content)

# 多線程
def  start_save_img(imgurl_list):
    for i in imgurl_list:
        th = threading.Thread(target=save_img,args=(i,))
        th.start() # 啟動(dòng)線程

# 多頁
def main():
    start_url = 'https://www.doutula.com/article/list?page='
    for i in range(1,50):
        start_html = get_html(start_url.format(i)) # 獲取外面的url
        get_img_html(start_html) # 獲取內(nèi)頁的url

if __name__ == '__main__': # 判斷文件入口
    main()

#main()

知識(shí)點(diǎn)總結(jié)

2017-04-12

python 2 編輯器：pycharm中文
python 庫：requests,lxml,bs4(beautifulsoup 4) 自帶 threading 多線程處理與控制
安裝使用pip install 安裝

python 開發(fā)網(wǎng)站：豆瓣，知乎，谷歌，果殼，YouTube。。。
優(yōu)點(diǎn)：代碼少,開發(fā)效率高，擁有大量第三方庫
方向：1.web后臺(tái)開發(fā)
2.自動(dòng)化運(yùn)維、測(cè)試
3.大數(shù)據(jù) 數(shù)據(jù)分析 挖掘 采集
4機(jī)器學(xué)習(xí)
5人工智能
6科學(xué)技術(shù)運(yùn)算
7 樹莓派 發(fā)射wifi 機(jī)械手臂
8 云計(jì)算。。。

課題：多線程爬取斗圖網(wǎng)站

1.網(wǎng)站禁止爬蟲
headers 添加
模擬瀏覽器
找到 user_agent 方法： f12 ---network---f5---name ---headers---user_agent

onerror="this.src='//img.doutula.com/production/uploads/image/2017/04/12/20170412979281_sEfqWn.jpg'"

插件安裝記錄

C:\Users\wang>pip list
You are using pip version 7.0.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
lxml (3.7.3)
pip (7.0.1)
setuptools (16.0)

C:\Users\wang>pip install beautifulsoup
You are using pip version 7.0.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting beautifulsoup
  Downloading BeautifulSoup-3.2.1.tar.gz
Installing collected packages: beautifulsoup
  Running setup.py install for beautifulsoup
Successfully installed beautifulsoup-3.2.1

C:\Users\wang>pip install beautifulsoup4
You are using pip version 7.0.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting beautifulsoup4
  Downloading beautifulsoup4-4.5.3-py2-none-any.whl (85kB)
    38% |████████████▎                   | 32kB 8.5kB/s eta 0:00:0
    43% |█████████████▉                  | 36kB 9.5kB/s eta 0:00:
    48% |███████████████▍                | 40kB 10kB/s eta 0:00
    52% |█████████████████               | 45kB 10kB/s eta 0:0
    57% |██████████████████▌             | 49kB 11kB/s eta 0
    62% |████████████████████            | 53kB 11kB/s eta
    67% |█████████████████████▌          | 57kB 11kB/s et
    72% |███████████████████████         | 61kB 12kB/s e
    76% |████████████████████████▋       | 65kB 12kB/s
    81% |██████████████████████████▏     | 69kB 12kB
    86% |███████████████████████████▊    | 73kB 22k
    91% |█████████████████████████████▎  | 77kB 2
    96% |██████████████████████████████▉ | 81kB
    100% |████████████████████████████████| 86k
 20kB/s
Installing collected packages: beautifulsoup4
Successfully installed beautifulsoup4-4.5.3

C:\Users\wang>pip install requests
You are using pip version 7.0.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting requests
  Using cached requests-2.13.0-py2.py3-none-any.whl
Installing collected packages: requests
Successfully installed requests-2.13.0

C:\Users\wang>pip list
You are using pip version 7.0.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
BeautifulSoup (3.2.1)
beautifulsoup4 (4.5.3)
lxml (3.7.3)
pip (7.0.1)
requests (2.13.0)
setuptools (16.0)

C:\Users\wang>