airbnb

有朋友要爬airbnb國(guó)內(nèi)各大城市房源數(shù)據(jù)寫(xiě)論文，正好有空閑時(shí)間，所以就寫(xiě)了下，詳情頁(yè)用的是JS加載所以在獲取詳情頁(yè)的時(shí)候采用了selenium模塊,最下面代碼展示了selenium獲取加載后網(wǎng)頁(yè)結(jié)構(gòu)的寫(xiě)法，有興趣的往下拉。

在搜索欄輸入成時(shí)候，出現(xiàn)以下url，所以將需要的城市寫(xiě)成列表添加進(jìn)紅框位置即可切換城市

將該頁(yè)面往下拉，翻頁(yè)標(biāo)簽，獲取翻頁(yè)標(biāo)簽內(nèi)的數(shù)字，并使用max函數(shù)獲取的最大值，這樣就得到了最大頁(yè)面的值

獲取詳情頁(yè)的url

訪(fǎng)問(wèn)這些詳情url發(fā)現(xiàn)是用JS加載進(jìn)來(lái)的，所以這里我們需要用到selenium+phantomjs去模擬瀏覽的形式去獲取加載后的HTML，這里有個(gè)小技巧如果使用selenium獲取加載后的HTML速度較慢，所以將加載完畢后的HTML保存到本地，用本地去調(diào)試抓取詳情頁(yè)。

分析HTML后，發(fā)現(xiàn)需要的詳情信息在div#details里，其中通過(guò)div.col-md-9還分為幾個(gè)信息塊，這些信息塊不是每個(gè)url都會(huì)展示，所以需要通過(guò).about-this-listing-section-title去判斷前面的塊標(biāo)簽，再選取相應(yīng)的數(shù)據(jù)抽取策略（下圖是HTML標(biāo)簽特征）

HTML標(biāo)簽特征

好了，簡(jiǎn)單的分析就到這詳細(xì)具體請(qǐng)看下面代碼

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) \
        AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36",
    "Connection": "keep-alive",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Accept-Language": "zh-CN,zh;q=0.8"
}
city_tuple = 'Shanghai', 'beijing', 'chengdu', 'chongqing', \
             'hangzhou', 'guangzhou', 'nanjing', 'suzhou', \
             'xian', 'qingdao'
for city_name in city_tuple:
    city_url = 'https://zh.airbnb.com/s/{}--China?cdn_cn=1&s_tag=Ic3cziVn&allow_override%5B%5D='.format(city_name)
    response = requests.get(city_url, headers=headers)
    # print(response.text)
    soup = BeautifulSoup(response.content, 'lxml')
    soup.encode('utf-8')
    # print(soup)
    info_area = soup.select('div.cardGrid_o1rkka div a')
    href_list = [_.get("href") for _ in info_area]
    # print(href_list)
    all_room_info_url_list = ['https://zh.airbnb.com/' + _ for _ in href_list]

    page_num_list = soup.select('div > ul.buttonList_11hau3k > li')
    max_page = max([int(_.get_text().strip()) for _ in page_num_list if _.get_text().strip().isdigit()])
    # print(max_page)
    for pg in range(1, max_page):
        city_url_pg = city_url + '& section_offset = {}'.format(pg)
        response = requests.get(city_url_pg, headers=headers)
        info_area = soup.select('div.cardGrid_o1rkka div a')
        href_list = [_.get("href") for _ in info_area]
        room_info_url_list = ['https://zh.airbnb.com/' + _ for _ in href_list]
        all_room_info_url_list.extend(room_info_url_list)
    # 城市及該城市下所有url
    print(city_name, all_room_info_url_list)

    # 詳情頁(yè)
    for info_url in all_room_info_url_list:
        print('詳情頁(yè)url', info_url)
        selenium_ua = headers["User-Agent"]
        options = dict(DesiredCapabilities.PHANTOMJS)
        options["phantomjs.page.settings.userAgent"] = selenium_ua
        browser = webdriver.PhantomJS(desired_capabilities=options)
        browser.get(info_url)
        browser.implicitly_wait(60)
        soup = BeautifulSoup(browser.page_source, 'lxml')
        browser.close()

        room_sorce_list = soup.select('div#details div.col-md-9')
        # print(len(room_sorce_list))

        # 評(píng)分
        star_rate = soup.select_one('div.star-rating')
        if star_rate is not None:
            star_rate = star_rate.get('content')
        print(star_rate)
        # 每日價(jià)格
        price = soup.select_one('span#book-it-price-string')
        if price is not None:
            price = price.get_text()
        print(price)

        section_title_list = [_.get_text() for _ in soup.select('.about-this-listing-section-title')]
        #       # 取出披露的項(xiàng)目列表
        # print(section_title_list)

        for idx_num, idx_title in enumerate(section_title_list):
            print(idx_num, idx_title)
            if idx_title == '商務(wù)差旅':
                for i in room_sorce_list[idx_num].select('span > span'):
                    # 商務(wù)差旅
                    print(i.get_text())
            if idx_title == '房源':
                for i in room_sorce_list[idx_num].select('div.bottom-spacing-2'):
                    # 房源信息
                    print(i.get_text())
            elif idx_title == '便利設(shè)施':
                print('**可使用設(shè)施**')
                for i in room_sorce_list[idx_num].select('strong'):
                    # 可使用設(shè)施
                    print(i.get_text())
                print('**不可使用設(shè)施**')
                for i in room_sorce_list[idx_num].select('del'):
                    # 不可使用設(shè)施
                    print(i.get_text())
            elif idx_title == '收費(fèi)相關(guān)' or idx_title == '相關(guān)收費(fèi)':
                for i in room_sorce_list[idx_num].select('div.bottom-spacing-2'):
                    # 收費(fèi)相關(guān)
                    print(i.get_text())
            elif idx_title == '床位安排':
                for i in room_sorce_list[idx_num].select('div.text-muted'):
                    # 床位安排
                    print(i.get_text())
            elif idx_title == '《房屋守則》':
                for i in room_sorce_list[idx_num].select('div.col-sm-12'):
                    # 《房屋守則》
                    print(i.get_text())
            elif idx_title == '預(yù)訂取消政策':
                for i in room_sorce_list[idx_num].select('p'):
                    # 預(yù)訂取消政策
                    print(i.get_text())
            elif idx_title == '安全設(shè)施':
                for i in room_sorce_list[idx_num].select('span'):
                    # 安全設(shè)施
                    print(i.get_text())
            elif idx_title == '可訂狀態(tài)':
                for i in room_sorce_list[idx_num].select('div.bottom-spacing-2')[:-1]:
                    # 可訂狀態(tài)
                    print(i.get_text())