PHP cURL 多線程並發抓取完整指南：核心 API 與三大實戰場景

cURL 多線程原理

`curl_multi_*` 系列函數實現單進程多連接並發，通過事件循環機制同時管理多個 cURL 句柄，顯著提升批量抓取效率。

封裝通用並發函數

function Curl_http(array $urls, int $timeout = 10): array {
    $mh = curl_multi_init();
    $conn = [];
    $start = microtime(true);
    
    // 初始化所有句柄
    foreach ($urls as $k => $url) {
        $conn[$k] = curl_init($url);
        curl_setopt_array($conn[$k], [
            CURLOPT_TIMEOUT => $timeout,
            CURLOPT_USERAGENT => 'Mozilla/5.0 (compatible; MSIE 5.01; Windows NT 5.0)',
            CURLOPT_MAXREDIRS => 7,
            CURLOPT_HEADER => 0,
            CURLOPT_FOLLOWLOCATION => 1,
            CURLOPT_RETURNTRANSFER => 1,
        ]);
        curl_multi_add_handle($mh, $conn[$k]);
    }
    
    // 事件循環
    $active = null;
    do { $mrc = curl_multi_exec($mh, $active); } 
    while ($mrc == CURLM_CALL_MULTI_PERFORM);
    
    while ($active && $mrc == CURLM_OK) {
        if (curl_multi_select($mh) != -1) {
            do { $mrc = curl_multi_exec($mh, $active); }
            while ($mrc == CURLM_CALL_MULTI_PERFORM);
        }
    }
    
    // 收集結果
    $res = $header = [];
    foreach ($urls as $k => $url) {
        $res[$k] = curl_multi_getcontent($conn[$k]);
        $header[$k] = curl_getinfo($conn[$k]);
        curl_multi_remove_handle($mh, $conn[$k]);
        curl_close($conn[$k]);
    }
    curl_multi_close($mh);
    
    return [
        'diff_time' => microtime(true) - $start,
        'return' => $res,
        'header' => $header
    ];
}

實戰場景一：並發抓取頁面存文件

$urls = ['https://www.jb51.net/', 'https://www.google.com/', 'https://www.example.com/'];
$fp = fopen('/test.txt', 'a');
$mh = curl_multi_init();

foreach ($urls as $i => $url) {
    $conn[$i] = curl_init($url);
    curl_setopt_array($conn[$i], [
        CURLOPT_FILE => $fp,           // 直接寫入文件句柄
        CURLOPT_HEADER => 0,
        CURLOPT_CONNECTTIMEOUT => 60,
    ]);
    curl_multi_add_handle($mh, $conn[$i]);
}

do { curl_multi_exec($mh, $active); } while ($active);

foreach ($urls as $i => $url) {
    curl_multi_remove_handle($mh, $conn[$i]);
    curl_close($conn[$i]);
}
curl_multi_close($mh);
fclose($fp);

實戰場景二：內存緩衝後寫入（可控內容處理）

// 關鍵差異：使用 CURLOPT_RETURNTRANSFER
curl_setopt($conn[$i], CURLOPT_RETURNTRANSFER, true);

do { curl_multi_exec($mh, $active); } while ($active);

foreach ($urls as $i => $url) {
    $data = curl_multi_getcontent($conn[$i]); // 獲取字符串
    fwrite($fp, $data);                        // 自定義處理後寫入
    // 此處可做內容過濾、編碼轉換、關鍵詞提取等
}

實戰場景三：並發下載文件

$urls = ['https://site.com/5w.zip', ...];
$save_to = './home/';
$mh = curl_multi_init();

foreach ($urls as $i => $url) {
    $file = $save_to . basename($url);
    if (is_file($file)) continue;  // 斷點續傳基礎
    
    $conn[$i] = curl_init($url);
    $fp[$i] = fopen($file, 'w');
    curl_setopt_array($conn[$i], [
        CURLOPT_FILE => $fp[$i],
        CURLOPT_HEADER => 0,
        CURLOPT_CONNECTTIMEOUT => 60,
    ]);
    curl_multi_add_handle($mh, $conn[$i]);
}

do { curl_multi_exec($mh, $active); } while ($active);

foreach ($urls as $i => $url) {
    curl_multi_remove_handle($mh, $conn[$i]);
    curl_close($conn[$i]);
    fclose($fp[$i]);
}
curl_multi_close($mh);

進階優化技巧

• POST 大數據：`curl_setopt($ch, CURLOPT_HTTPHEADER, ['Expect:']);` 避免 100-continue 延遲
• 連接復用：保持 `$mh` 長期運行，動態增減句柄
• 錯誤處理：檢查 `curl_error($conn[$k])` 和 `curl_getinfo($conn[$k], CURLINFO_HTTP_CODE)`
• 限流控制：配合 `usleep()` 或令牌桶算法控制併發數

總結

cURL 多線程是 PHP 高性能網絡抓取的核心技術。掌握 `curl_multi_exec` 事件循環模型，結合場景化參數調優，可構建穩健的爬蟲、API 聚合、文件下載系統。