PHP cURL 多線程並發抓取：curl_multi 實戰與大文件下載

痛點：PHP 單線程阻塞

PHP 運行時單線程，循環 curl_exec 串行抓取效率極低。curl_multi 系列函數提供**事件驅動並發**，單進程實現多連接並行。

核心流程

1. curl_multi_init() 創建多句柄
2. 逐個 curl_init($url) 創建子句柄，設置選項
3. curl_multi_add_handle() 掛載子句柄
4. 循環 curl_multi_exec() 驅動狀態機，直到無活動句柄
5. curl_multi_remove_handle() 卸載、curl_close() 釋放
6. curl_multi_close() 銷毀多句柄

場景一：並發抓取頁面並直接寫入文件

<?php
$urls = [
    'https://www.sina.com.cn/',
    'https://www.sohu.com/',
    'https://www.163.com/',
];
$saveTo = '/tmp/fetch_results.txt';
$fp = fopen($saveTo, 'a');

$mh = curl_multi_init();
$handles = [];

foreach ($urls as $i => $url) {
    $ch = curl_init($url);
    curl_setopt_array($ch, [
        CURLOPT_USERAGENT      => 'Mozilla/5.0 (compatible; PHP Bot)',
        CURLOPT_HEADER         => 0,
        CURLOPT_CONNECTTIMEOUT => 10,
        CURLOPT_TIMEOUT        => 30,
        CURLOPT_FILE           => $fp, // 直接寫入文件句柄
    ]);
    curl_multi_add_handle($mh, $ch);
    $handles[$i] = $ch;
}

$active = null;
do {
    $mrc = curl_multi_exec($mh, $active);
} while ($mrc === CURLM_CALL_MULTI_PERFORM || $active);

foreach ($handles as $ch) {
    curl_multi_remove_handle($mh, $ch);
    curl_close($ch);
}
curl_multi_close($mh);
fclose($fp);

場景二：並發抓取到變量，再統一處理（推薦）

<?php
$urls = [ /* ... */ ];
$mh = curl_multi_init();
$handles = [];

foreach ($urls as $i => $url) {
    $ch = curl_init($url);
    curl_setopt_array($ch, [
        CURLOPT_USERAGENT       => 'Mozilla/5.0',
        CURLOPT_HEADER          => 0,
        CURLOPT_RETURNTRANSFER  => 1, // 關鍵：返回字串而非直接輸出
        CURLOPT_CONNECTTIMEOUT  => 10,
        CURLOPT_TIMEOUT         => 30,
    ]);
    curl_multi_add_handle($mh, $ch);
    $handles[$i] = $ch;
}

$active = null;
do {
    curl_multi_exec($mh, $active);
} while ($active);

// 獲取結果
$results = [];
foreach ($handles as $i => $ch) {
    $results[$i] = curl_multi_getcontent($ch);
    curl_multi_remove_handle($mh, $ch);
    curl_close($ch);
}
curl_multi_close($mh);

// 後續處理：存 DB、解析、寫文件...
foreach ($results as $url => $html) {
    // parse($html);
    file_put_contents("/tmp/{$url}.html", $html);
}

關鍵選項解析

大文件並發下載進階模式

function multiDownload(array $urlMap, string $saveDir, int $concurrency = 10): void {
    // $urlMap: ['filename' => 'https://...']
    $mh = curl_multi_init();
    $active = 0;
    $queue = $urlMap;
    $running = [];

    while ($queue || $running) {
        // 啟動新任務至並發上限
        while (count($running) < $concurrency && $queue) {
            [$file, $url] = array_shift($queue);
            $ch = curl_init($url);
            $fp = fopen($saveDir . '/' . $file, 'wb');
            curl_setopt_array($ch, [
                CURLOPT_FILE           => $fp,
                CURLOPT_HEADER         => 0,
                CURLOPT_CONNECTTIMEOUT => 30,
                CURLOPT_TIMEOUT        => 300,
                CURLOPT_FOLLOWLOCATION => 1,
                CURLOPT_USERAGENT      => 'Downloader/1.0',
            ]);
            curl_multi_add_handle($mh, $ch);
            $running[(int)$ch] = $fp;
        }

        // 驅動執行
        do {
            $mrc = curl_multi_exec($mh, $active);
        } while ($mrc === CURLM_CALL_MULTI_PERFORM);

        // 檢查完成項
        while ($info = curl_multi_info_read($mh)) {
            $ch = $info['handle'];
            $fp = $running[(int)$ch];
            fclose($fp);
            curl_multi_remove_handle($mh, $ch);
            curl_close($ch);
            unset($running[(int)$ch]);
        }

        // 無活動時短暫休眠避免空轉
        if ($active === 0 && $queue) {
            usleep(10000); // 10ms
        }
    }
    curl_multi_close($mh);
}

效能調優關鍵參數

現代替代：Guzzle + Promise / Swoole / OpenSwoole

• Guzzle Promise： PSR-7/18 兼容、中介軟體、異步協程風格
• Swoole/OpenSwoole： 真協程並發、毫秒級調度、支援 HTTP/2、WebSocket
• amphp/artax： 純 PHP 異步 HTTP 客戶端

curl_multi 是原生擴展無依賴的並發利器，理解其事件循環機制對掌握異步編程思想大有裨益。生產環境高並發場景建議遷移至 Swoole/Workerman 等協程框架。