爬蟲功能
獲取上交所和深交所所有股票的名稱和交易信息
保存到文件中

技術(shù)路線
requests-bs4-re

候選數(shù)據(jù)網(wǎng)站
新浪股票：http://finance.sina.com.cn/stock/
百度股票：https://gupiao.baidu.com/stock
選取原則
股票信息保存在HTML頁面中，不是js動態(tài)生成的，沒有robots協(xié)議的限制

數(shù)據(jù)網(wǎng)站的確定
獲取股票列表;
東方財富網(wǎng)：http://quote.eastmoney.com/stocklist.html
獲取個股信息：
百度股票：https://gupiao.baidu.com/stock/
單個股票：https://gupiao.baidu.com/stock/sz002439.html

程序的結(jié)構(gòu)設(shè)計
從東方財富網(wǎng)獲取股票列表
根據(jù)股票列表逐個到百度股票獲取個股信息
將獲取的信息保存到文件中

個股信息采用鍵值對維護

main()

import re,request
from bs4 import BeaytifulSoup
import traceback

def getHTMLText(url):
    return ""

def getStockList(lst,stockURL):
    return ""

def getStockInfo(lst,stockURL,fpath):
    return ""

def main():
    stock_list_url="http://quote.eastmoney.com/stocklist.html"
    stock_info_url="https://gupiao.baifu.com/stock/"
    output_file="D://BaiduStockInfo.txt"
    slist=[]
    getStockList(slist,stock_list_url)
    getStockInfo(slist,stock_info_url,output_file)

main()

geteHTMLText()

def getHTMLText(url):
    try:
        r=requests.get(url)
        r.raise_for_status()
        r.encoding=r.apparent_encoding
    except:
        return ""

getStockList()

def getStockList(lst,stockURL):
    html=getHTMLText(stockURL)
    soup=BeautifulSoup(html,'html.parser')
    a=soup('a')
    for i in a:
        try:
            href=i.attrs['href']
            lst.append(re.findall(r"[s][zh]\d{6}",href)[0])
        except:
            continue       

getStockInfo()

def getStockInfo(lst, stockURL, fpath):
    for stock in lst:
        url = stockURL + stock + ".html"
        html = getHTMLText(url)
        try:
            if html=="":
                continue
            infoDict = {}
            soup = BeautifulSoup(html, 'html.parser')
            stockInfo = soup.find('div',attrs={'class':'stock-bets'})
 
            name = stockInfo.find_all(attrs={'class':'bets-name'})[0]
            infoDict.update({'股票名稱': name.text.split()[0]})
             
            keyList = stockInfo.find_all('dt')
            valueList = stockInfo.find_all('dd')
            for i in range(len(keyList)):
                key = keyList[i].text
                val = valueList[i].text
                infoDict[key] = val
             
            with open(fpath, 'a', encoding='utf-8') as f:
                f.write( str(infoDict) + '\n' )
        except:
            traceback.print_exc()
            continue

實例優(yōu)化——提高用戶體驗

速度提高：編碼識別的優(yōu)化
r.apparent_encoding需要分析文本，運行較慢，可輔助人工分析

def getHTMLText(url, code="utf-8"):
    try:
        r = requests.get(url)
        r.raise_for_status()
        r.encoding = code
        return r.text
    except:
        return ""
 
def getStockList(lst, stockURL):
    html = getHTMLText(stockURL, "GB2312")
    soup = BeautifulSoup(html, 'html.parser') 
    a = soup.find_all('a')
    for i in a:
        try:
            href = i.attrs['href']
            lst.append(re.findall(r"[s][hz]\d{6}", href)[0])
        except:
            continue

體驗提高：增加動態(tài)進度顯示