從零構建網頁爬蟲：核心原理、BFS 策略與 Java 完整實作

爬蟲核心原理

網頁爬蟲本質是**有向圖遍歷**：網頁 = 節點，超連結 = 邊。核心流程：

1. 種子 URL 入隊
2. 取出 URL → 下載頁面 → 解析提取新 URL → 入隊
3. 去重 (已訪問集合) → 重複步驟 2
4. 滿足停止條件 (數量、深度、時間) 結束

遍歷策略：廣度優先 (BFS) vs 深度優先 (DFS)

**主流爬蟲均採用 BFS + 優先級佇列** (PageRank、更新頻率、權重評分)。

Java 完整實作 (教學級)

依賴

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.14</version>
</dependency>
<dependency>
    <groupId>org.htmlparser</groupId>
    <artifactId>htmlparser</artifactId>
    <version>2.1</version>
</dependency>

1. URL 隊列管理 (執行緒安全)

public class UrlQueue {
    private final Queue<String> unvisited = new ConcurrentLinkedQueue<>();
    private final Set<String> visited = ConcurrentHashMap.newKeySet();
    private final int maxSize;

    public UrlQueue(int maxSize) { this.maxSize = maxSize; }

    public void addSeed(String... seeds) {
        for (String s : seeds) addUnvisited(s);
    }

    public synchronized void addUnvisited(String url) {
        if (url == null || url.isBlank()) return;
        url = normalize(url);
        if (!visited.contains(url) && !unvisited.contains(url)) {
            unvisited.offer(url);
        }
    }

    public String poll() {
        String url = unvisited.poll();
        if (url != null) visited.add(url);
        return url;
    }

    public boolean isEmpty() { return unvisited.isEmpty(); }
    public int visitedCount() { return visited.size(); }
    public boolean isFinished() { return visitedCount() >= maxSize; }

    private String normalize(String url) {
        // 移除 fragment、統一協議大小寫、補全相對路徑等
        return url.split("#")[0].trim();
    }
}

2. 頁面下載器 (HttpClient)

public class PageDownloader {
    private final CloseableHttpClient client;
    private final String userAgent = "Mozilla/5.0 (compatible; MyBot/1.0)";

    public PageDownloader() {
        this.client = HttpClients.custom()
            .setDefaultRequestConfig(RequestConfig.custom()
                .setConnectTimeout(5000)
                .setSocketTimeout(10000)
                .build())
            .setUserAgent(userAgent)
            .build();
    }

    public String download(String url) {
        HttpGet get = new HttpGet(url);
        try (CloseableHttpResponse resp = client.execute(get)) {
            int status = resp.getStatusLine().getStatusCode();
            if (status != HttpStatus.SC_OK) return null;
            HttpEntity entity = resp.getEntity();
            return entity != null ? EntityUtils.toString(entity, "UTF-8") : null;
        } catch (IOException e) {
            return null; // 記錄日誌
        }
    }

    public void close() throws IOException { client.close(); }
}

3. 連結提取器 (HtmlParser)

public class LinkExtractor {
    public Set<String> extract(String baseUrl, String html) {
        Set<String> links = new HashSet<>();
        try {
            Parser parser = new Parser(html);
            parser.setEncoding("UTF-8");
            
            NodeFilter aFilter = new TagNameFilter("A");
            NodeFilter frameFilter = new NodeFilter() {
                @Override public boolean accept(Node node) {
                    return node.getText().startsWith("frame src=");
                }
            };
            
            NodeList list = parser.extractAllNodesThatMatch(new OrFilter(aFilter, frameFilter));
            for (int i = 0; i < list.size(); i++) {
                Node node = list.elementAt(i);
                String href = null;
                if (node instanceof LinkTag) {
                    href = ((LinkTag) node).getLink();
                } else { // frame
                    String text = node.getText();
                    int start = text.indexOf("src=") + 5;
                    int end = text.indexOf(' ', start);
                    if (end == -1) end = text.indexOf('>', start);
                    href = text.substring(start, end).replace("\"", "");
                }
                if (href != null) {
                    links.add(resolve(baseUrl, href));
                }
            }
        } catch (ParserException e) {
            // 記錄
        }
        return links;
    }

    private String resolve(String base, String href) {
        try {
            return new URL(new URL(base), href).toString();
        } catch (MalformedURLException e) {
            return null;
        }
    }
}

4. 爬蟲主控制器

public class SimpleCrawler {
    private final UrlQueue queue;
    private final PageDownloader downloader;
    private final LinkExtractor extractor;
    private final LinkFilter filter; // 只爬特定域名

    public SimpleCrawler(String[] seeds, int maxPages, LinkFilter filter) {
        this.queue = new UrlQueue(maxPages);
        this.queue.addSeed(seeds);
        this.downloader = new PageDownloader();
        this.extractor = new LinkExtractor();
        this.filter = filter;
    }

    public void start() {
        while (!queue.isEmpty() && !queue.isFinished()) {
            String url = queue.poll();
            if (url == null) continue;
            
            System.out.println("[" + queue.visitedCount() + "] 爬取: " + url);
            String html = downloader.download(url);
            if (html == null) continue;
            
            // 儲存/處理 html...
            savePage(url, html);
            
            // 提取新連結
            Set<String> links = extractor.extract(url, html);
            for (String link : links) {
                if (filter.accept(link)) {
                    queue.addUnvisited(link);
                }
            }
            
            // 禮貌延遲
            sleep(100);
        }
        downloader.close();
    }

    private void savePage(String url, String html) {
        // 寫入檔案、資料庫、索引...
    }

    private void sleep(long ms) {
        try { Thread.sleep(ms); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
    }

    public static void main(String[] args) {
        LinkFilter filter = url -> url.startsWith("https://example.com/");
        new SimpleCrawler(new String[]{"https://example.com/"}, 1000, filter).start();
    }
}

生產級關鍵增強

法律與道德合規

• 遵守 robots.txt (User-agent, Disallow, Crawl-delay)
• 設定合理 Crawl-delay，不造成目標伺服器過載
• 識別自我：User-Agent 包含聯繫方式
• **不爬取**需登入、個人隱私、版權受保護內容
• 遵守網站服務條款 (ToS) 及當地法律 (GDPR、個資法等)

總結

爬蟲核心 = **BFS + 去重 + 下載 + 解析 + 存儲**。教學版幾百行代碼即可跑通流程；生產級需在**分散式架構、智能調度、反反爬、合規治理**上深耕。Java 生態擁有 HttpClient、Jsoup、WebMagic、Nutch 等成熟框架，**建議直接基於 WebMagic/Nutch 二次開發**，而非從零造輪子。