python中解析網(wǎng)頁內(nèi)容基本步驟

1. 使用BeautifulSoup解析網(wǎng)頁
   Soup = BeautifulSoup(html, 'lxml')
2. 描述要爬取得東西在哪里
   =Soup.select(‘路徑’)
3. 從標簽中獲得需要的信息，按一定格式裝在數(shù)據(jù)容器中（字典的列表），便于查詢
   <p>Something</p>
   [{title=Something, rate=4.0},{title=Something, rate=4.0},{title=Something, rate=4.0},{title=Something, rate=4.0}]

兩種不同的路徑描述方式

• CSS Selector:
  例如：body>div.main-content>ul>li:nth-child(1)>img
• XPATH
  例如：/html/body/div[2]/ul/li(1)/img

作業(yè)代碼

from bs4 import BeautifulSoup
info= []
#讀取本地HTML文件并找到要爬取的片段
with open('E:/工作盤Workshop/a3-編程練習/Python Practice/0824/1.2/index.html','r',) as web_data:
    Soup = BeautifulSoup(web_data,'lxml')
    images = Soup.select('body > div:nth-of-type(1)> div > div.col-md-9 > div:nth-of-type(2) > div > div > img')
    titles = Soup.select('body > div:nth-of-type(1) > div > div.col-md-9 > div:nth-of-type(2) > div > div > div.caption > h4 > a')
    prices = Soup.select('body > div:nth-of-type(1) > div > div.col-md-9 > div:nth-of-type(2) > div > div > div.caption > h4.pull-right')
    votes = Soup.select('body > div:nth-of-type(1) > div > div.col-md-9 > div:nth-of-type(2) > div > div > div.ratings > p.pull-right')
    stars = Soup.select('body > div:nth-of-type(1) > div > div.col-md-9 > div:nth-of-type(2) > div > div > div.ratings > p:nth-of-type(2)')
    #print(images,titles,prices,votes,stars,sep='\n------------------\n')

#從片段中提取有效數(shù)據(jù)，每組數(shù)據(jù)生成一個字典，再把字典放入列表
for title,image,price,vote,star in zip(titles,images,prices,votes,stars):
    data = {
        'title':title.get_text(),
        'price':price.get_text(),
        'star': len(star.find_all("span", "glyphicon glyphicon-star")),
        'vote':vote.get_text()[:-8],
        'image':image.get('src')
    }
    info.append(data)

#因列表較長，用一個函數(shù)逐條打印各元素
def print_lol(the_list):
    for each_item in the_list:
        print(each_item)

print_lol(info)

輸出效果

1.jpg

遺留問題

• Chrome瀏覽器中copy selector得到的路徑中有一處錯誤，手工更正后才能找到元素。元素實際位置是body > div:nth-of-type(1)下面，瀏覽器復制出來的路徑卻是body > div:nth-of-type(2)。
• 能否找到更高效的方法把-child替換為-of-type，手工修改確實很浪費時間。