讀者最好對 Python 和爬蟲有一些了解，所以關(guān)于 Scrapy 一些信息比如如何安裝的方法等我就不介紹了。

案例

爬下知乎某個話題下面的精華問答，包括推薦回答，回答者，問題簡介，問題鏈接。爬完數(shù)據(jù)存在本地文件。

新建工程

首先新建一個 Scrapy 工程，在工程目錄命令行執(zhí)行

scrapy startproject zhihu

之后生成的文件有幾個是需要我們知道的
items.py 是定義我們的數(shù)據(jù)結(jié)構(gòu)的
pipelines.py 是處理數(shù)據(jù)的，一般就在里面存數(shù)據(jù)庫，或者文件
settings.py 是配置爬蟲信息，比如 USER_AGENT 還有 ITEM_PIPELINES 這些
./spiders 文件夾下面需要我們建立我們的爬蟲程序

編碼

在 ./spiders 新建一個 zhihu_questions_spider.py 的文件，定義一個 ZhihuQustionsSpider 類 繼承自 CrawlSpider，這是已經(jīng)編寫好的代碼

from scrapy.spiders import Spider
from scrapy.selector import Selector
from scrapy.spiders import Rule, CrawlSpider
from scrapy.linkextractors import LinkExtractor

from zhihu.items import ZhihuQuestionItem


class ZhihuQustionsSpider(CrawlSpider):

    name = "zhihu_qustions_spider"
    allowed_domains = ["www.zhihu.com"]
    start_urls = ['https://www.zhihu.com/topic/19571921/top-answers?page=1']
    rules = (Rule(
        LinkExtractor(
            allow=(r'https://www\.zhihu\.com/topic/19571921/top-answers\?page=\d{2}$')),
        callback='parse_item',
        follow=True), )

    def parse_item(self, response):
        selector = Selector(response)
        item = ZhihuQuestionItem()
        question_name = selector.xpath(
            '//a[@class="question_link"][1]/text()').extract()
        question_url = selector.xpath(
            '//a[@class="question_link"][1]/@href').extract()
        best_answer_author = selector.xpath(
            '//a[@class="author-link"][1]/text()').extract()
        best_answer_summary = selector.xpath(
            '//div[@class="zh-summary summary clearfix"][1]/text()').extract()
        item['question_name'] = [n for n in question_name]
        item['best_answer_author'] = [n for n in best_answer_author]
        item['question_url'] = [
            'https://www.zhihu.com' + n for n in question_url]
        item['best_answer_summary'] = [n for n in best_answer_summary]
        #print (item)
        yield item

name 是爬蟲名字，后面會用到
allowed_domains 是允許爬的域名
start_urls 是開始爬的地址
rules 定義什么樣的鏈接是我們需要接著爬的，里面的 callback 就是我們的解析函數(shù)
parse_item 是我們解析重要的函數(shù)，這里我使用 xpath 解析，不了解可以去網(wǎng)上找教程學(xué)習(xí)下，里面的元素信息可以通過chrome的開發(fā)者調(diào)試模式拿到。
items.py

import scrapy
class ZhihuQuestionItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    question_name = scrapy.Field()
    question_url = scrapy.Field()
    best_answer_author = scrapy.Field()
    best_answer_summary = scrapy.Field()
    pass

**pipelines.py **

import codecs
import json

class ZhihuPipeline(object):

    def __init__(self):
        self.file = codecs.open('zhihu_qustions.txt',
                                mode='wb', encoding='utf-8')
    def process_item(self, item, spider):
        line = ''
        for i in range(len(item['question_name'])):
            question_name = {'question_name': str(
                item['question_name'][i]).replace('\n', '')}
            try:
                question_url = {'question_url': item['question_url'][i]}
            except IndexError:
                question_url = {'question_url': ""}
            try:
                best_answer_author = {'best_answer_author': item['best_answer_author'][i]}
            except IndexError:
                best_answer_author = {'best_answer_author': ""}
            try:
                best_answer_summary = {'best_answer_summary': str(
                item['best_answer_summary'][i]).replace('\n', '')}
            except IndexError:
                best_answer_summary = {'best_answer_summary': ""}
            line = line + json.dumps(question_name, ensure_ascii=False)
            line = line + json.dumps(best_answer_author, ensure_ascii=False)
            line = line + json.dumps(best_answer_summary, ensure_ascii=False)
            line = line + json.dumps(question_url, ensure_ascii=False) + '\n'
        self.file.write(line)

    def close_spider(self, spider):
        self.file.close()

初始化打開一個文件，然后在數(shù)據(jù)處理函數(shù) process_item 轉(zhuǎn)成 json 寫到文件里面。
settings.py 加上以下這些

USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
ITEM_PIPELINES = {
    'zhihu.pipelines.ZhihuPipeline': 300,
}
DOWNLOAD_DELAY = 1

DOWNLOAD_DELAY 是指每個鏈接請求的間隔，可以避免太頻繁的調(diào)用被封。

運行

在工程目錄執(zhí)行

scrapy crawl zhihu_qustions_spider

如果正常的話，就會生成 zhihu_qustions.txt 文件，大概是這樣的

shoot.jpeg

源碼我我就不傳GitHub了，上面基本都包括了