Urllib是python內(nèi)置的http請(qǐng)求庫(kù)，分為以下幾個(gè)模塊

• urllib.request：請(qǐng)求模塊
• urllib.error：url異常處理模塊
• urllib.parse：url解析模塊
• urllib.robotparser：robots.txt解析模塊

1.urllib.request

1.1 urllib.request.urlopen()

urllib.request.urlopen(url, data=None, [timeout,]*, ...)

• url：要打開(kāi)的連接
• data=None：附加數(shù)據(jù)，例如使用post方式的時(shí)候附加的數(shù)據(jù)
• timeout：超時(shí)時(shí)間

1.1.1 url

import urllib.request
response = urllib.request.urlopen('http://www.baidu.com')
print(response.read().decode('utf-8'))

• response.read()方法返回的是bytes類型數(shù)據(jù)，需要decode成相應(yīng)編碼的字符串
• 這是一個(gè)get請(qǐng)求方式

1.1.2 data

import urllib.request
import urllib.parse

data = bytes(urllib.parse.urlencode({'world':'hello'}), encoding='utf-8')
respon = urllib.request.urlopen('http://httpbin.org/post', data=data)
print(respon.read())

• 加入了data參數(shù)，這是一個(gè)post請(qǐng)求方式

1.1.3 timeout

import socket
import urllib.request
import urllib.error

try:
    respon = urllib.request.urlopen('http://www.baidu.com', timeout=1)
except urllib.error.URLError as e:
    if isinstance(e.reason, socket.timeout):
        print('Time Out')
try:
    respon = urllib.request.urlopen('http://www.baidu.com', timeout=0.01)
except urllib.error.URLError as e:
    if isinstance(e.reason, socket.timeout):
        print('Time Out')

1.2 響應(yīng)

1.2.1 響應(yīng)類型

import urllib.request
respon = urllib.request.urlopen('https://www.python.org')
print(type(respon))
================================================================================================
>> <class 'http.client.HTTPResponse'>

1.2.2 狀態(tài)碼與響應(yīng)頭

import urllib.request
respon = urllib.request.urlopen('http://www.python.org')
print(respon.status)
print(respon.getheaders())
print(respon.getheader('Server'))

• respon.status：獲取狀態(tài)碼
• respon.getheaders()：所有的響應(yīng)頭
• respon.getheader('Server')：獲取特定響應(yīng)頭

1.3 Request對(duì)象

1.3.1 使用Request對(duì)象發(fā)起請(qǐng)求

import urllib.request as rq
requ = rq.Request('http://www.baidu.com')
resp = rq.urlopen(requ)
print(resp.read().decode('utf-8'))

• 聲明一個(gè)Request對(duì)象
• 把Rquest對(duì)象傳入urllib.request.urlopen()中

1.3.2 使用Request對(duì)象發(fā)起請(qǐng)求并攜帶額外數(shù)據(jù)

from urllib import request, parse
url = 'http://httpbin.org/post'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; …) Gecko/20100101 Firefox/57.0'
}
dict = {
    'name': 'doggy'
}
data = bytes(parse.urlencode(dict), encoding='utf-8')
req = request.Request(url=url, data=data, headers=headers, method='POST')
resp = request.urlopen(req)
print(resp.read().decode('utf-8'))

1.3.2 使用Request對(duì)象添加頭信息

from urllib import request, parse
url = 'http://httpbin.org/post'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; …) Gecko/20100101 Firefox/57.0'
}
dict = {
    'name': 'doggy'
}
data = bytes(parse.urlencode(dict), encoding='utf-8')
req = request.Request(url=url, data=data, method='POST')
req.add_header(headers)
resp = request.urlopen(req)
print(resp.read().decode('utf-8'))

2. urllib.error

urllib.error模塊定義了三個(gè)錯(cuò)誤類：

• urllib.error.URLError
  • reason：出錯(cuò)原因
• urllib.error.HTTPError
  • code：出錯(cuò)碼
  • reason：出錯(cuò)原因
  • headers：http響應(yīng)頭
  • urllib.error.HTTPError是urllib.error.URLError的子類
• urllib.error.ContentTooShortError

3. urllib.parse

3.1 urllib.parse.urlencode()

urllib.parse.urlencode()用來(lái)把一個(gè)字典數(shù)據(jù)轉(zhuǎn)換成get請(qǐng)求的參數(shù)

from urllib.parse import urlencode
params = {
    'name': 'yindf',
    'age':22
}
base_url = 'http://www.baidu.com?'
url = base_url + urlencode(params)
print(url)
================================================================================================
>> http://www.baidu.com?name=yindf&age=22

4. urllib.robotparse