Scrapy簡介
Scrapy是一個為了爬取網站數據，提取結構性數據而編寫的應用框架。
Scrapy入門請看官方文檔： [ scrapy官方文檔 ](http://scrapy-
chs.readthedocs.io/zh_CN/1.0/intro/tutorial.html)
本爬蟲簡介
本爬蟲實現按分類爬取豆瓣電影信息，一次爬取一個分類，且自動切換代理池，防止ip在訪問過多過頻繁后無效。
分類如圖所示：
實現-scrapy中間件
scrapy基礎框架參考上面的官方教程，搭建好基礎框架后，本爬蟲特殊之處在于為了防止爬蟲被封，采用了輪換代理和agent的中間件。
agent輪換池：
簡單的寫一個user_agent_list來使得每次的agent不同，原理簡單，代碼如下：
class RotateUserAgentMiddleware(UserAgentMiddleware): #輪換代理agent
def __init__(self, user_agent=''):
self.user_agent = user_agent
def process_request(self, request, spider):
ua = random.choice(self.user_agent_list)
if ua:
#print '-----------------------Using user-agent:', ua, '------------------------'
request.headers.setdefault('User-Agent', ua)
# the default user_agent_list composes chrome,IE,firefox,Mozilla,opera,netscape
# for more user agent strings,you can find it in http://www.useragentstring.com/pages/useragentstring.php
user_agent_list = [ \
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1" \
"Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11", \
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6", \
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6", \
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1", \
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5", \
"Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5", \
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", \
"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", \
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3", \
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3", \
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3", \
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", \
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", \
"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3", \
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3", \
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24", \
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
]
ip輪換池：
采用了一位大神cocoakekeyu寫的中間件 Github地址
，并不認識他，但是為他點贊。代碼在這里不貼了，可以去Github看。
** “一個用于scrapy爬蟲的自動代理中間件。可自動抓取和切換代理，自定義抓取和切換規則。” **
**
**
**
**
實現-爬蟲實現
item.py
class DoubanItem(scrapy.Item):
movie_name = scrapy.Field()
movie_director = scrapy.Field()
movie_writer = scrapy.Field()
movie_starring = scrapy.Field()
movie_category = scrapy.Field()
movie_country = scrapy.Field()
#movie_language = scrapy.Field()
movie_date = scrapy.Field()
movie_time = scrapy.Field()
movie_star = scrapy.Field()
movie_5score = scrapy.Field()
movie_4score = scrapy.Field()
movie_3score = scrapy.Field()
movie_2score = scrapy.Field()
movie_1score = scrapy.Field()
movie_describe = scrapy.Field()
pass
看這item名都不用我解釋...
doubanlist_spider.py
先貼上代碼：
class doubanlistSpider(scrapy.Spider):
name = "doubanlist"
allowed_domains = ["movie.douban.com"]
start_urls = [
"https://movie.douban.com/tag/%E5%8A%A8%E7%94%BB"
]
def parse(self, response):
for href in response.xpath('//a[@class="nbg"]/@href'):
url = href.extract()
yield scrapy.Request(url, callback=self.parse_each_movie)
next_page = response.xpath('//span[@class="next"]/a/@href').extract()
if next_page:
print '--------------Finding next page: [%s] --------------------------', next_page
yield scrapy.Request(next_page[0], callback=self.parse)
else:
print '--------------There is no more page!--------------------------'
def parse_each_movie(self, response):
item = DoubanItem()
item['movie_name'] = response.xpath('//span[@property="v:itemreviewed"]/text()').extract()
item['movie_director'] = response.xpath('//a[@rel="v:directedBy"]/text()').extract()
item['movie_writer'] = response.xpath('//span[@class="attrs"][2]/a/text()').extract()
item['movie_starring'] = response.xpath('//a[@rel="v:starring"]/text()').extract()
item['movie_category'] = response.xpath('//span[@property="v:genre"]/text()').extract()
#item['movie_language'] = response.xpath('//*[@id="info"]').re(r'</span> (.*)<br>\n')[2]
item['movie_date'] = response.xpath('//span[@property="v:initialReleaseDate"]/text()').extract()
item['movie_time'] = response.xpath('//span[@property="v:runtime"]/text()').extract()
item['movie_star'] = response.xpath('//strong[@property="v:average"]/text()').extract()
item['movie_5score'] = response.xpath('//span[@class="rating_per"][1]/text()').extract()
item['movie_4score'] = response.xpath('//span[@class="rating_per"][2]/text()').extract()
item['movie_3score'] = response.xpath('//span[@class="rating_per"][3]/text()').extract()
item['movie_2score'] = response.xpath('//span[@class="rating_per"][4]/text()').extract()
item['movie_1score'] = response.xpath('//span[@class="rating_per"][5]/text()').extract()
item['movie_describe'] = response.xpath('//*[@id="link-report"]/span/text()').re(r'\S+')
check_item = response.xpath('//*[@id="info"]').re(r'</span> (.*)<br>\n')[1]
result = self.check_contain_chinese(check_item)
if result:
item['movie_country'] = response.xpath('//*[@id="info"]').re(r'</span> (.*)<br>\n')[1]
else:
item['movie_country'] = response.xpath('//*[@id="info"]').re(r'</span> (.*)<br>\n')[2]
yield item
def check_contain_chinese(self, check_str):
for ch in check_str.decode('utf-8'):
if u'\u4e00' <= ch <= u'\u9fff':
return True
return False
def parse(self, response)：從https://movie.douban.com/tag/%E5%8A%A8%E7%94%BB（某一特
定分類）開始，爬取20條本頁的電影，之后判定“下一頁”按鈕是否存在，如果存在則繼續爬取下一頁。
def parse_each_movie(self, response):對于每個電影詳細頁，爬取所需要的信息，全部使用xpath
中間一段是在爬取電影國家信息時，由于有不同情況的網頁（可能是新老頁面交替），需要不同處理，不然會爬到不正確的信息，xpath定位不準。
def check_contain_chinese：為了確定爬取的中文內容為中文字符串，需要進行判斷。
