初學(xué)Python,試著用requests + BeautifulSoup + MySQLdb抓取豆瓣圖書TOP250的各類數(shù)據(jù)同時存入數(shù)據(jù)庫。
1.目標(biāo)
url:
豆瓣圖書TOP250:https://book.douban.com/top250?start=0
一共有250個圖書，每一頁返回25條數(shù)據(jù)，start從0到最大225。
html:
圖1.png
每一頁的25條數(shù)據(jù)都在<td valign="top">里面，對應(yīng)有圖書名、出版社、評分(可選)、參評人數(shù)、一句話介紹等數(shù)據(jù)，目的就是把以上數(shù)據(jù)篩選出來存入MySQL中。
2.開始
先用命令創(chuàng)建一個MySQL數(shù)據(jù)庫,一定要設(shè)置默認(rèn)字符集為utf8,否則可能會導(dǎo)致無法插入中文數(shù)據(jù):
MacBook-Pro:~ Tan$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.19 MySQL Community Server (GPL)
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> CREATE DATABASE NEWDATABASE DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
douban.py用于網(wǎng)絡(luò)請求和數(shù)據(jù)解析
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests
from bs4 import BeautifulSoup
import sys
import time
from dbManager import DBManager
reload(sys)
sys.setdefaultencoding( "utf-8" )
# 獲取圖書TOP250
class Douban():
"""docstring for Douban"""
def __init__(self):
self.baseurl = 'https://book.douban.com/top250?start='
self.agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'
self.headers = {'User-Agent':self.agent}
self.page = 0
self.maxPage = 250 - 25
self.db = DBManager()
#拼接當(dāng)前url
def getCurrentUrl(self):
url = self.baseurl + str(self.page)
print 'url:%s' % url
return url
#TOP250數(shù)據(jù)都在<td valign="top">里面
def has_valign_but_no_width(self,tag):
return tag.has_attr('valign') and not tag.has_attr('width')
#獲取每一頁的數(shù)據(jù)
def loadPage(self):
url = self.getCurrentUrl()
result = requests.get(url,headers=self.headers).content
soup = BeautifulSoup(result,'html.parser', from_encoding='utf-8')
content = soup.find_all(self.has_valign_but_no_width)
#遍歷,解析有用的數(shù)據(jù)
for item in content:
title = item.a['title']
publishingHouse = item.p.string
price = publishingHouse.split('/')[-1]
ratingNums = item.find('span',class_ = 'rating_nums').string
ratingPeoples = item.find('span',class_ = 'pl').string[1:-1].strip()
inq = ''
if item.find('span',class_ = 'inq'):
inq = item.find('span',class_ = 'inq').string
print 'title:%s\npublishingHouse:%s\nprice:%s\nratingNums:%s\nratingPeoples:%s\ninq:%s\n' % (title,publishingHouse,price,ratingNums,ratingPeoples,inq)
#插入數(shù)據(jù)庫
self.db.insert(title,publishingHouse,price,ratingNums,ratingPeoples,inq)
#抓取完一頁 休息3s
time.sleep(3)
#抓取下一頁
self.page += 25
if self.page <= self.maxPage:
self.loadPage()
else:
# 爬完就關(guān)閉數(shù)據(jù)庫
self.db.closeDB()
if __name__ == '__main__':
douban = Douban()
douban.loadPage()
dbManager用于數(shù)據(jù)存儲
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import MySQLdb
import sys
reload(sys)
sys.setdefaultencoding( "utf-8" )
class DBManager():
"""docstring for DBManager"""
SQL_CREATE = '''CREATE TABLE IF NOT EXISTS DOUBANBOOK (id int unsigned not null AUTO_INCREMENT primary key,
title varchar(20) not null,
publishingHouse varchar(50),
price varchar(10),
ratingNums varchar(10),
ratingPeoples varchar(20),
inq varchar(20))'''
SQL_INSERT = '''INSERT INTO DOUBANBOOK (title,publishingHouse,price,ratingNums,ratingPeoples,inq) VALUES (%s,%s,%s,%s,%s,%s)'''
def __init__(self):
self.host = 'localhost'
self.dbName = '數(shù)據(jù)庫名'
self.userName = 'root'
self.password = '密碼'
self._db = MySQLdb.connect(host = self.host,
user = self.userName,
passwd = self.password,
db = self.dbName,
charset="utf8")
self._cursor = self._db.cursor()
self._execute(sql=self.SQL_CREATE)
def _execute(self,params=None,sql=''):
if params:
self._cursor.execute(sql,params)
else:
self._cursor.execute(sql)
def insert(self,title,publishingHouse,price,ratingNums,ratingPeoples,inq):
try:
self._execute(params=(title,publishingHouse,price,ratingNums,ratingPeoples,inq),sql=self.SQL_INSERT)
self._db.commit()
except Exception as e:
print 'db error %s' % e
self._db.rollback()
def closeDB(self):
self._db.close()
3.最后
運行douban.py:
url:https://book.douban.com/top250?start=0
title:追風(fēng)箏的人
publishingHouse:[美] 卡勒德·胡賽尼 / 李繼宏 / 上海人民出版社 / 2006-5 / 29.00元
price: 29.00元
ratingNums:8.9
ratingPeoples:283124人評價
inq:為你，千千萬萬遍
title:小王子
publishingHouse:[法] 圣埃克蘇佩里 / 馬振聘 / 人民文學(xué)出版社 / 2003-8 / 22.00元
price: 22.00元
ratingNums:9.0
ratingPeoples:225146人評價
inq:獻(xiàn)給長成了大人的孩子們
title:圍城
publishingHouse:錢鍾書 / 人民文學(xué)出版社 / 1991-2 / 19.00
price: 19.00
ratingNums:8.9
ratingPeoples:188666人評價
inq:對于“人艱不拆”四個字最徹底的違抗
title:解憂雜貨店
publishingHouse:[日] 東野圭吾 / 李盈春 / 南海出版公司 / 2014-5 / 39.50元
price: 39.50元
ratingNums:8.6
ratingPeoples:233024人評價
inq:一碗精心熬制的東野牌雞湯，拒絕很難
圖2.png
數(shù)據(jù)庫結(jié)構(gòu):
+-----------------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+------------------+------+-----+---------+----------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| title | varchar(20) | NO | | NULL | |
| publishingHouse | varchar(50) | YES | | NULL | |
| price | varchar(10) | YES | | NULL | |
| ratingNums | varchar(10) | YES | | NULL | |
| ratingPeoples | varchar(20) | YES | | NULL | |
| inq | varchar(20) | YES | | NULL | |
+-----------------+------------------+------+-----+---------+----------------+
接下來，準(zhǔn)備用這個數(shù)據(jù)庫數(shù)據(jù)配合flask_restful做一個簡單的api，返回給App使用。
