防盜鏈介紹:基礎(chǔ)防盜鏈主要是針對(duì)客戶端請(qǐng)求過程中所攜帶的一些關(guān)鍵信息來驗(yàn)證請(qǐng)求的合法性， 比如客戶端請(qǐng)求IP，請(qǐng)求URL中攜帶的referer。優(yōu)點(diǎn)是規(guī)則簡單，配置和使用都很方便，缺點(diǎn)是防盜鏈所依賴的驗(yàn)證信息很多都是可以偽造的，因此此類防盜鏈可靠性較低。

創(chuàng)建工程: scrapy startproject tutorial

2.創(chuàng)建蜘蛛 scrapy genspider imgSpider

3.創(chuàng)建爬取的Item

#圖片下載管道
class ImageItem(scrapy.Item):
   image_urls = scrapy.Field()
   images = scrapy.Field()
   image_name = scrapy.Field()

4.蜘蛛代碼:網(wǎng)上很多用中間件的方式偽造referer,但是經(jīng)過嘗試是不可以的中間件在去訪問資源的時(shí)候，拿到的referer和圖片的鏈接一樣這樣的話圖片會(huì)被重定向，具體看spider代碼

import scrapy
from tutorial.items import ImageItem

class ImgspiderSpider(scrapy.Spider):
    name = 'imgSpider'
    allowed_domains = ['www.mm131.com']
    start_urls =  ['http://www.mm131.com/xinggan/',
                'http://www.mm131.com/qingchun/',
                'http://www.mm131.com/xiaohua/',
                'http://www.mm131.com/chemo/',
                'http://www.mm131.com/qipao/',
                 'http://www.mm131.com/mingxing/'
                ]


    def parse(self, response):
        list = response.css(".list-left dd:not(.page)")
        for image in list:
            image_name = image.css("a::text").extract_first()
            image_url = image.css("a::attr(href)").extract_first()
            image_url2 = str(image_url)
            print(image_url2)
            next_page = response.css(".page-en:nth-last-child(2)::attr(href)").extract_first()
            if next_page is not None:
                yield response.follow(next_page,callback=self.parse)

            #下載圖片
            yield scrapy.Request(image_url2,callback=self.downloadImage)

    def downloadImage(self,response):
        item = ImageItem()
        item['image_name'] = response.css(".content h5::text").extract_first()

        item['image_urls'] = response.css(".content-pic img::attr(src)").extract()
        print('---------------image_urls---------',item['image_urls'])
        #防盜鏈:referer 從那個(gè)頁面過來 沒有來源就圖片就會(huì)被重定向 解決辦法在請(qǐng)求頭中添加 headers={"referer":referer}
        # 解決防盜鏈的最根本的就是告訴訪問的資源的請(qǐng)求來自本站
        item['referer'] = response.url
        yield item
        next_url = response.css(".page-ch:last-child::attr(href)").extract_first()
        if next_url is not None:
            yield response.follow(next_url,callback=self.downloadImage,dont_filter=True)

5.編寫pipline處理下載的圖片分組

class ImagePipline(ImagesPipeline):
    def get_media_requests(self, item, info):
        for image_url in item["image_urls"]:
            print('-------------------image_url------------------------%s',image_url)
            yield Request(image_url,meta={'name':item['image_name']},headers={'referer':item['referer']})

    #重命名的功能 重寫此功能可以得到自己想要文件名稱 否則就是uuid的隨機(jī)字符串
    def file_path(self, request, response=None, info=None):
        #圖片名稱
        img_name = request.url.split('/')[-1]
        #圖片分類的名稱
        name = request.meta['name']
        #處理特殊字符串
        name = re.sub(r'[？\\*|“<>:/()0123456789]','',name)
        #分文件夾存儲(chǔ)
        filename = u'{0}/{1}'.format(name,img_name)
        return filename

    def item_completed(self, results, item, info):
        image_paths = [x['path'] for ok,x in results if ok]
        #上面的表達(dá)式等于
        # for ok,x in results:
        #     if ok:
        #         print(x['path'])
        if not image_paths:
            raise DropItem('Item contains no images')
        item['image_urls'] = image_paths
        return item

.如果你看到下面的圖片代表你成功了

屏幕快照 2019-01-02 19.46.05.png