PHP 正則提取網頁超連結：URL 解析、相對路徑轉換與遞迴採集

資料採集首步常需提取頁面所有連結。本文提供健壯的正則模式、相對/絕對 URL 標準化、多層遞迴採集與編碼處理完整方案。

核心提取函式

<?php
/**
 * 從 HTML 提取所有超連結
 * @param string $baseUrl 基礎 URL (用於相對路徑轉換)
 * @param string $html HTML 內容
 * @return array ['links' => [...], 'texts' => [...], 'full' => [...]]
 */
function extractLinks(string $baseUrl, string $html): array {
    // 模式：匹配 text，支援單雙引號、屬性順序不固定
    $pattern = '/<a\s+[^>]*href\s*=\s*([\'"])(.*?)\\1[^>]*>(.*?)<\/a>/is';
    
    if (!preg_match_all($pattern, $html, $matches)) {
        return ['links' => [], 'texts' => [], 'full' => []];
    }
    
    $urls = $matches[2];
    $texts = $matches[3];
    $fullTags = $matches[0];
    
    $parsedBase = parse_url($baseUrl);
    $baseHost = $parsedBase['scheme'] . '://' . $parsedBase['host'];
    $basePath = rtrim(dirname($parsedBase['path'] ?? '/'), '/') . '/';
    
    $normalizedLinks = [];
    foreach ($urls as $url) {
        $url = trim($url);
        if ($url === '' || $url === '#' || strpos($url, 'javascript:') === 0) continue;
        
        // 絕對 URL
        if (preg_match('~^https?://~i', $url)) {
            $normalizedLinks[] = $url;
        }
        // 協定相對 URL (//example.com/path)
        elseif (strpos($url, '//') === 0) {
            $normalizedLinks[] = $parsedBase['scheme'] . ':' . $url;
        }
        // 根相對路徑 (/path)
        elseif ($url[0] === '/') {
            $normalizedLinks[] = $baseHost . $url;
        }
        // 相對路徑 (path/to/page)
        else {
            $normalizedLinks[] = $baseHost . $basePath . ltrim($url, './');
        }
    }
    
    // 去重
    $normalizedLinks = array_values(array_unique($normalizedLinks));
    
    return [
        'links' => $normalizedLinks,
        'texts' => array_values(array_filter($texts, fn($t) => trim($t) !== '')),
        'full' => $fullTags
    ];
}

獲取頁面內容 (含編碼自動處理)

<?php
function fetchPage(string $url): ?string {
    $ch = curl_init($url);
    curl_setopt_array($ch, [
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_FOLLOWLOCATION => true,
        CURLOPT_MAXREDIRS => 5,
        CURLOPT_CONNECTTIMEOUT => 10,
        CURLOPT_TIMEOUT => 30,
        CURLOPT_ENCODING => '', // 自動解壓 gzip/deflate/br
        CURLOPT_SSL_VERIFYPEER => true,
        CURLOPT_USERAGENT => 'Mozilla/5.0 (compatible; PHP Link Extractor)',
    ]);
    
    $content = curl_exec($ch);
    $errno = curl_errno($ch);
    curl_close($ch);
    
    if ($errno) return null;
    
    // 編碼偵測轉 UTF-8
    $encoding = mb_detect_encoding($content, ['UTF-8', 'GBK', 'BIG5', 'ISO-8859-1'], true);
    if ($encoding && $encoding !== 'UTF-8') {
        $content = iconv($encoding, 'UTF-8//IGNORE', $content);
    }
    
    return $content;
}

多層遞迴採集器

<?php
class LinkCrawler {
    private $visited = [];
    private $maxDepth;
    private $delayMs;
    private $sameDomainOnly;
    
    public function __construct(int $maxDepth = 2, int $delayMs = 500, bool $sameDomainOnly = true) {
        $this->maxDepth = $maxDepth;
        $this->delayMs = $delayMs;
        $this->sameDomainOnly = $sameDomainOnly;
    }
    
    public function crawl(string $startUrl): array {
        $this->visited = [];
        return $this->crawlRecursive($startUrl, 1);
    }
    
    private function crawlRecursive(string $url, int $depth): array {
        if ($depth > $this->maxDepth) return [];
        if (isset($this->visited[$url])) return [];
        $this->visited[$url] = true;
        
        $html = fetchPage($url);
        if (!$html) return [];
        
        $result = extractLinks($url, $html);
        $allContent = $this->extractTextContent($html);
        
        // 遞迴採集子連結
        if ($depth < $this->maxDepth) {
            foreach ($result['links'] as $link) {
                if ($this->sameDomainOnly && !$this->isSameDomain($url, $link)) continue;
                
                usleep($this->delayMs * 1000); // 禮貌延遲
                $childResult = $this->crawlRecursive($link, $depth + 1);
                $allContent .= $childResult;
            }
        }
        
        return $allContent;
    }
    
    private function extractTextContent(string $html): string {
        // 移除 script/style
        $html = preg_replace('@<script[^>]*>.*?</script>@is', '', $html);
        $html = preg_replace('@<style[^>]*>.*?</style>@is', '', $html);
        // 移除標籤
        $text = strip_tags($html);
        // 提取中文
        preg_match_all('/[\x{4e00}-\x{9fff}]+/u', $text, $matches);
        return implode('，', $matches[0]);
    }
    
    private function isSameDomain(string $url1, string $url2): bool {
        $host1 = parse_url($url1, PHP_URL_HOST);
        $host2 = parse_url($url2, PHP_URL_HOST);
        return $host1 === $host2;
    }
}

// 使用
$crawler = new LinkCrawler(2, 1000, true);
$content = $crawler->crawl('https://example.com/start-page');
file_put_contents('collected_text.txt', $content);

進階正則模式對比

DOMDocument 替代方案 (推薦複雜頁面)

<?php
function extractLinksDom(string $baseUrl, string $html): array {
    libxml_use_internal_errors(true);
    $dom = new DOMDocument();
    $dom->loadHTML('<?xml encoding="UTF-8"?>' . $html);
    libxml_clear_errors();
    
    $xpath = new DOMXPath($dom);
    $links = $xpath->query('//a[@href]');
    
    $result = ['links' => [], 'texts' => [], 'full' => []];
    $parsedBase = parse_url($baseUrl);
    $baseHost = $parsedBase['scheme'] . '://' . $parsedBase['host'];
    
    foreach ($links as $link) {
        $href = $link->getAttribute('href');
        $text = trim($link->textContent);
        
        if ($href === '' || $href === '#') continue;
        
        // 同樣的 URL 正規化邏輯...
        $normalized = normalizeUrl($baseUrl, $href);
        $result['links'][] = $normalized;
        $result['texts'][] = $text;
        $result['full'][] = $dom->saveHTML($link);
    }
    
    return $result;
}

注意事項與最佳實踐

• 遵守 robots.txt：採集前檢查 User-agent: * 規則
• 頻率控制：usleep(500000) 0.5秒延遲，避免被封
• 去重策略：URL 正規化後 array_unique()，考慮資料庫唯一索引
• 錯誤處理：記錄失敗 URL、HTTP 狀態碼、重試機制
• 法律合規：僅採集公開資料、尊重版權、遵守服務條款