嗨嘍~大家好呀，這里是魔王吶 ? ~!
知識點:
爬蟲基本流程
抓包工具的使用
開發(fā)環(huán)境:
python 3.10 運行代碼
pycharm 輔助敲代碼
requests 第三方模塊
reqable
微信客戶端
案例實現(xiàn)流程:
一. 需求(思路)分析
找到數(shù)據(jù)的來源
https://mp.weixin.qq.com/mp/profile_ext?action=getmsg&__biz=MzkwOTI2MjM0NQ==&f=json&offset=10&count=10&is_ok=1&scene=124&uin=MjQwODIxNTMyMw==&key=f2bc285be4cea20f508e3183cf22028899478d6c6cd992a5fca68b37abbd2f67cfc36330e6207cb4d60037d436adbb8b5796a26d079539fd38b6737a5e89a3b80fb45bd3f29f86fac95f75a62c39a5fbbdc4cb92108740ab094cd297b6ad5850a8621b12b8ab3634574fe9e98ab0253c9ec79e9215f655bece9642bc78d9612f&pass_ticket=NQQv+JhXtqD/xxBpc74ykqMwmRHbEvRHPSvlKXdD9VS67OBNPGiaFD+l2QEJIhaG/ZW25Jtp7evC92qtD7tLVA==&wxtoken=&appmsg_token=1249_5DoqtKeztxPNkr0O83A6cr7n9IsXeFAjjn8zjA~~&x5=0&f=json
二. 具體代碼實現(xiàn)
發(fā)送請求
獲取數(shù)據(jù)
-
解析數(shù)據(jù)
非結(jié)構(gòu)化數(shù)據(jù) 網(wǎng)頁源代碼
結(jié)構(gòu)化數(shù)據(jù) json數(shù)據(jù)
保存數(shù)據(jù)
代碼展示
導入模塊
import requests # 發(fā)送請求的第三方庫
import json
import re
模擬偽裝
headers = {
"host": "mp.weixin.qq.com",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 NetType/WIFI MicroMessenger/7.0.20.1781(0x6700143B) WindowsWechat(0x6309080f) XWEB/8461 Flue",
"x-requested-with": "XMLHttpRequest",
"accept": "*/*",
"sec-fetch-site": "same-origin",
"sec-fetch-mode": "cors",
"sec-fetch-dest": "empty",
"referer": "https",
"accept-encoding": "gzip, deflate, br",
"accept-language": "zh-CN,zh;q=0.9",
"cookie": "wap_sid2=CJvmqfwIEooBeV9ISlZFcmIyd2lIdU5EeHFLMGdvQTB5eHdrQldFeENNRzREM2RXRkozbVNabzlsa2pFc1lSVS1WMjlRQVZQZGtKQnhmMjdMeGtvcFBULXhnQ2tIMW1tUlR2MGM1cGhzMU16eW1DWkVmZFpZT2JiYVZsMExPNU9aV2p6NU1sVWJnTGdiNFNBQUF+MMm+i6wGOA1AlU4="
}
請求鏈接
url = 'https://mp.weixin.qq.com/mp/profile_ext?action=getmsg&__biz=MzkwOTI2MjM0NQ==&f=json&offset=20&count=10&is_ok=1&scene=124&uin=MjQwODIxNTMyMw==&key=098370c6c4230f2782cb126aad5ceee0079fda057ac38504504c5c4e237bd74afa5aa169e7db8882e0509aa1c04ef4c92ddf193e767f26b0c3d0697b2822575f337ae4a2556c04a8bfc1bcbb4715d8d2b25711985de0493188b0a34dcb01f8bcbaa67c7bf757527bf6674f72ea36e47a53280a25064df4317fa94a59236eb81b&pass_ticket=NQQv+JhXtqD/xxBpc74ykqMwmRHbEvRHPSvlKXdD9VRrkCQePU8FiLP9JBjS4XXiYNn3/yrKsy2bYkfSepRFlQ==&wxtoken=&appmsg_token=1249_JNc5MRSwDmbSu2vnkNrxa_RwVFe_A9i_hvejmA~~&x5=0&f=json'
- 發(fā)送請求
response = requests.get(url=url, headers=headers)
- 獲取數(shù)據(jù)
# general_msg_list
json_data = response.json()
- 解析數(shù)據(jù)
general_msg = json_data['general_msg_list']
general_msg_list = json.loads(general_msg)
count = 0
for msg in general_msg_list['list']:
title = msg['app_msg_ext_info']['title']
content_url = msg['app_msg_ext_info']['content_url']
- 繼續(xù)發(fā)送請求
print(title, content_url)
if content_url:
text_content = requests.get(content_url, headers=headers).text
# print(text_content)
# css / xpath / re
# .*?: 代替任意字符
img_list = re.findall('<img class="rich_pages wxw-img".*?data-src="(.*?)".*?>', text_content)
print(img_list)
for img in img_list:
- 保存數(shù)據(jù)
img_data = requests.get(img).content
with open(f'img/{count}.jpg', mode='wb') as f:
f.write(img_data)
count += 1
尾語
最后感謝你觀看我的文章吶~本次航班到這里就結(jié)束啦 ??
希望本篇文章有對你帶來幫助 ??，有學習到一點知識~
躲起來的星星??也在努力發(fā)光，你也要努力加油（讓我們一起努力叭）。
