Python urllib/urllib2/httplib 抓取網頁完整範例：Request、Proxy、Cookie、重導向處理

urllib2 (Python 2) / urllib.request (Python 3) 提供強大的 HTTP 客戶端功能，本文整理常用模式。

基本抓取

import urllib2
f = urllib2.urlopen(url, timeout=5).read()

Request 物件：Headers、POST、Method

req = urllib2.Request(url, headers={'User-Agent': 'Mozilla/5.0'})
# POST
req = urllib2.Request(url, data=urllib.urlencode(params), headers=headers)
# 自訂 Method
req.get_method = lambda: 'PUT'

Proxy 代理設定

proxy = urllib2.ProxyHandler({'http': 'https://proxy:8080'})
opener = urllib2.build_opener(proxy, urllib2.HTTPHandler)
urllib2.install_opener(opener)

Cookie 處理

import cookielib
cookie = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie))
opener.open(login_url, urllib.urlencode({'username':'u', 'password':'p'}))
for c in cookie: print(c.name, c.value)

重導向追蹤

import httplib
httplib.HTTPConnection.debuglevel = 1
opener = urllib2.build_opener()
f = opener.open(req)
print(f.url)      # 最終 URL
print(f.headers)  # 響應頭

完整 Demo 結構

if __name__ == '__main__':
    use_urllib2()
    get_request()
    get_request_direct()
    use_proxy()
    hand_cookie()