您现在的位置：程式碼之家 >> 編程語言 > >> .NET 網頁程式設計 >> ASP.NET 基礎 >> ASP.NET 基礎

C# 爬蟲實戰：兩種攜帶 Cookie 進行需登入頁面資料採集技巧 C# 爬蟲實戰：兩種攜帶 Cookie 進行需登入頁面資料採集技巧

編輯：ASP.NET 基礎

核心原理

需登入頁面的存取控制本質是**伺服器驗證 HTTP Request Header 中的 Cookie (或 SessionID)**。採集時只需在 HttpWebRequest 中附上合法 Cookie，伺服器即視為已登入使用者並回傳內容。

方案一：直接注入已知 Cookie

適用情境：已透過瀏覽器開發者工具取得有效 Cookie (如 username、sessionid)，且 Cookie 有效期較長。

public void Collect(Hashtable cookies)
{
    string url = "https://target.com/protected-page";
    string host = "https://target.com";

    byte[] data = Encoding.UTF8.GetBytes(""); // POST body 若有

    var req = (HttpWebRequest)WebRequest.Create(url);
    req.Method = "POST";
    req.ContentType = "application/json;charset=utf-8";
    req.ContentLength = data.Length;

    // 關鍵：建立 CookieContainer 並加入 Cookie
    var cc = new CookieContainer();
    foreach (DictionaryEntry entry in cookies)
    {
        cc.Add(new Uri(host), new Cookie(entry.Key.ToString(), entry.Value.ToString()));
    }
    req.CookieContainer = cc;

    // 送出請求
    using (var stream = req.GetRequestStream())
        stream.Write(data, 0, data.Length);

    // 接收回應
    using (var resp = (HttpWebResponse)req.GetResponse())
    using (var respStream = resp.GetResponseStream())
    using (var reader = new StreamReader(respStream, Encoding.UTF8))
    {
        string html = reader.ReadToEnd();
        // 解析 html...
    }
}

// 呼叫範例
var ht = new Hashtable { {"username", "myAccount"}, {"sessionid", "abc123"} };
Collect(ht);

方案二：模擬登入取得 CookieContainer 再採集

適用情境：需處理驗證碼、動態 Token、或 Cookie 短效需即時登入。

protected void Page_Load(object sender, EventArgs e)
{
    var cc = new CookieContainer();

    // 步驟 1：模擬登入 POST
    string loginUrl = "https://target.com/login";
    string postData = "username=admin&password=123456"; // 依實際表單欄位調整
    byte[] postBytes = Encoding.UTF8.GetBytes(postData);

    var loginReq = (HttpWebRequest)WebRequest.Create(loginUrl);
    loginReq.Method = "POST";
    loginReq.ContentType = "application/x-www-form-urlencoded";
    loginReq.CookieContainer = cc; // 同一 Container 接收 Set-Cookie
    loginReq.ContentLength = postBytes.Length;

    using (var s = loginReq.GetRequestStream())
        s.Write(postBytes, 0, postBytes.Length);

    using (var loginResp = (HttpWebResponse)loginReq.GetResponse()) 
    { 
        // 僅為觸發 Set-Cookie 存入 cc，內容可忽略
    }

    // 步驟 2：攜帶同一 CookieContainer 存取受保護頁面
    string targetUrl = "https://target.com/protected-page";
    var targetReq = (HttpWebRequest)WebRequest.Create(targetUrl);
    targetReq.Method = "GET";
    targetReq.CookieContainer = cc; // 關鍵：重用含登入 Cookie 的 Container

    using (var targetResp = (HttpWebResponse)targetReq.GetResponse())
    using (var stream = targetResp.GetResponseStream())
    using (var reader = new StreamReader(stream, Encoding.GetEncoding("gb2312"))) // 依目標編碼
    {
        string html = reader.ReadToEnd();
        Response.Write(html); // 或進一步解析
    }
}