PHP cURL 多線程併發抓取：curl_multi 完整封裝與三大應用場景實戰

PHP cURL 多線程併發抓取：curl_multi 完整封裝與三大應用場景實戰

PHP 單線程特性下，curl_multi 是實現併發 HTTP 請求的核心方案。本文提供通用封裝函數，並演示三大場景：併發抓取頁面、併發下載文件、結果寫入文件，附性能優化參數詳解。

通用併發抓取封裝

function curlMultiRequest(array $urls, int $timeout = 10): array {
    $mh = curl_multi_init();
    $handles = [];
    $start = microtime(true);

    foreach ($urls as $key => $url) {
        $ch = curl_init($url);
        curl_setopt_array($ch, [
            CURLOPT_TIMEOUT => $timeout,
            CURLOPT_CONNECTTIMEOUT => 5,
            CURLOPT_USERAGENT => 'Mozilla/5.0 (compatible; PHP-cURL-Multi/1.0)',
            CURLOPT_MAXREDIRS => 7,
            CURLOPT_FOLLOWLOCATION => true,
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_HEADER => false,
            CURLOPT_SSL_VERIFYPEER => false, // 內網/測試環境
        ]);
        curl_multi_add_handle($mh, $ch);
        $handles[$key] = $ch;
    }

    // 執行併發
    $active = null;
    do {
        $mrc = curl_multi_exec($mh, $active);
    } while ($mrc === CURLM_CALL_MULTI_PERFORM);

    while ($active && $mrc === CURLM_OK) {
        if (curl_multi_select($mh) !== -1) {
            do {
                $mrc = curl_multi_exec($mh, $active);
            } while ($mrc === CURLM_CALL_MULTI_PERFORM);
        }
    }

    // 收集結果
    $results = [];
    foreach ($handles as $key => $ch) {
        $results[$key] = [
            'content' => curl_multi_getcontent($ch),
            'info' => curl_getinfo($ch),
            'error' => curl_error($ch),
            'errno' => curl_errno($ch),
        ];
        curl_multi_remove_handle($mh, $ch);
        curl_close($ch);
    }
    curl_multi_close($mh);

    $results['_meta'] = [
        'total_time' => microtime(true) - $start,
        'count' => count($urls),
    ];
    return $results;
}

場景一：併發抓取多個頁面並保存到文件

$urls = [
    'https://www.bkjia.com/',
    'https://www.google.com/',
    'https://www.example.com/'
];
$saveFile = '/tmp/pages.txt';
$fp = fopen($saveFile, 'w');

$mh = curl_multi_init();
$handles = [];
foreach ($urls as $i => $url) {
    $ch = curl_init($url);
    curl_setopt_array($ch, [
        CURLOPT_USERAGENT => 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)',
        CURLOPT_HEADER => false,
        CURLOPT_FILE => $fp,  // 直接寫入文件句柄
        CURLOPT_CONNECTTIMEOUT => 60,
    ]);
    curl_multi_add_handle($mh, $ch);
    $handles[$i] = $ch;
}

do { curl_multi_exec($mh, $active); } while ($active);

foreach ($handles as $ch) {
    curl_multi_remove_handle($mh, $ch);
    curl_close($ch);
}
curl_multi_close($mh);
fclose($fp);

場景二：併發抓取存變量再寫文件 (可後處理)

$urls = ['https://a.com/', 'https://b.com/'];
$saveFile = '/tmp/pages2.txt';
$fp = fopen($saveFile, 'w');

$mh = curl_multi_init();
$handles = [];
foreach ($urls as $i => $url) {
    $ch = curl_init($url);
    curl_setopt_array($ch, [
        CURLOPT_USERAGENT => '...',
        CURLOPT_HEADER => false,
        CURLOPT_RETURNTRANSFER => true,  // 返回字符串
        CURLOPT_CONNECTTIMEOUT => 60,
    ]);
    curl_multi_add_handle($mh, $ch);
    $handles[$i] = $ch;
}

do { curl_multi_exec($mh, $active); } while ($active);

foreach ($handles as $i => $ch) {
    $data = curl_multi_getcontent($ch);
    fwrite($fp, "=== $urls[$i] ===\n$data\n\n");
    curl_multi_remove_handle($mh, $ch);
    curl_close($ch);
}
curl_multi_close($mh);
fclose($fp);

場景三：併發下載大文件 (斷點續傳基礎)

$urls = [
    'https://example.com/large1.zip',
    'https://example.com/large2.zip',
];
$saveDir = '/home/downloads/';

$mh = curl_multi_init();
$handles = [];
$fps = [];

foreach ($urls as $i => $url) {
    $fileName = basename($url);
    $filePath = $saveDir . $fileName;
    if (is_file($filePath)) continue; // 簡單去重

    $ch = curl_init($url);
    $fp = fopen($filePath, 'w');
    curl_setopt_array($ch, [
        CURLOPT_USERAGENT => '...',
        CURLOPT_FILE => $fp,
        CURLOPT_HEADER => false,
        CURLOPT_CONNECTTIMEOUT => 60,
    ]);
    curl_multi_add_handle($mh, $ch);
    $handles[$i] = $ch;
    $fps[$i] = $fp;
}

do { curl_multi_exec($mh, $active); } while ($active);

foreach ($handles as $i => $ch) {
    curl_multi_remove_handle($mh, $ch);
    curl_close($ch);
    fclose($fps[$i]);
}
curl_multi_close($mh);

關鍵優化參數

性能對比 (抓取 3 個站點)

• 串行：~3.2 秒 (各 ~1 秒)
• curl_multi：~1.1 秒 (並發，取決於最慢請求)
• 提速：3x+

常見坑

• curl_multi_select 返回 -1 時需 usleep(10000) 避免 CPU 100%
• 大量並發 (>100) 時需分批，受限於文件描述符、端口、目標服務器限流
• SSL 驗證失敗設置 CURLOPT_SSL_VERIFYPEER => false (僅測試/內網)
• 內存峰值 = 併發數 × 平均響應體大小，大文件必須用 CURLOPT_FILE 流式寫入

掌握 curl_multi，配合生產者-消費者模型 (Redis/RabbitMQ 隊列)，可構建高性能爬蟲、API 聚合網關、文件分發系統。