crawler4j 實戰：輕量級多線程網絡爬蟲抓取自如租房數據完整教程

crawler4j 實戰：輕量級多線程網絡爬蟲抓取自如租房數據完整教程

crawler4j 是 Java 生態中輕量級、易用的開源網絡爬蟲框架，幾分鐘即可構建多線程爬蟲。本文結合 Jsoup 解析 HTML 及 JavaCSV 導出數據，完整演示抓取自如租房網（https://sh.ziroom.com/z/nl/）租房信息的全流程。

項目依賴

<dependencies>
    <!-- crawler4j 核心 -->
    <dependency>
        <groupId>edu.uci.ics</groupId>
        <artifactId>crawler4j</artifactId>
        <version>4.2</version>
    </dependency>

    <!-- Jsoup HTML 解析 -->
    <dependency>
        <groupId>org.jsoup</groupId>
        <artifactId>jsoup</artifactId>
        <version>1.8.3</version>
    </dependency>

    <!-- JavaCSV 導出 CSV -->
    <dependency>
        <groupId>net.sourceforge.javacsv</groupId>
        <artifactId>javacsv</artifactId>
        <version>2.0</version>
    </dependency>
</dependencies>

核心組件設計

1. 統計信息收集類 CrawlStat

public class CrawlStat {
    private long totalLinks = 0;
    private int totalProcessedPages = 0;
    private long totalTextSize = 0;

    // getter / setter / 增量方法 略
    public void incProcessedPages() { totalProcessedPages++; }
    public void incTotalLinks(int count) { totalLinks += count; }
    public void incTotalTextSize(int count) { totalTextSize += count; }
}

2. 自定義爬蟲 ZiroomCrawler 繼承 WebCrawler

public class ZiroomCrawler extends WebCrawler {

    private static final String URL_PREFIX = "https://sh.ziroom.com/z/nl/";
    // 過濾靜態資源
    private static final Pattern FILTERS = Pattern.compile(".*(\\.(css|js|bmp|gif|jpe?g|ico|png|tiff?|mid|mp2|mp3|mp4|wav|avi|mov|mpeg|ram|m4v|pdf|rm|smil|wmv|swf|wma|zip|rar|gz))$");

    private final File csvFile = new File("data/crawl/ziroom.csv");
    private final File linkFile = new File("data/crawl/link.csv");
    private final CrawlStat stat = new CrawlStat();

    @Override
    public boolean shouldVisit(Page referringPage, WebURL url) {
        String href = url.getURL().toLowerCase();
        return !FILTERS.matcher(href).matches() && href.startsWith(URL_PREFIX);
    }

    @Override
    public void visit(Page page) {
        String url = page.getWebURL().getURL();
        System.out.println("-----------爬取路徑：" + url);
        stat.incProcessedPages();

        if (page.getParseData() instanceof HtmlParseData) {
            HtmlParseData htmlParseData = (HtmlParseData) page.getParseData();
            String html = htmlParseData.getHtml();
            Document doc = Jsoup.parse(html);

            // 提取列表項
            Elements items = doc.select("li.clearfix");
            for (Element item : items) {
                String img = item.select(".img img").attr("src");
                Element txt = item.select(".txt").first();
                String addr = txt.select("h3 a").text() + "," + txt.select("h4 a").text() + "," + txt.select(".detail").text();
                String desc = txt.select("p").text();
                String price = item.select(".price").text();

                writeToCsv(img, price, addr, desc);
            }

            // 統計鏈接
            Set<WebURL> links = htmlParseData.getOutgoingUrls();
            stat.incTotalLinks(links.size());
            writeLinksToCsv(links);
        }
    }

    private void writeToCsv(String img, String price, String addr, String desc) {
        try (CsvWriter writer = new CsvWriter(new FileWriter(csvFile, true), ',')) {
            writer.write(img);
            writer.write(price);
            writer.write(addr);
            writer.write(desc);
            writer.endRecord();
        } catch (IOException e) { e.printStackTrace(); }
    }

    private void writeLinksToCsv(Set<WebURL> links) {
        try (CsvWriter writer = new CsvWriter(new FileWriter(linkFile, true), ',')) {
            for (WebURL u : links) writer.write(u.getURL());
            writer.endRecord();
        } catch (IOException e) { e.printStackTrace(); }
    }

    @Override
    public Object getMyLocalData() { return stat; }

    @Override
    public void onBeforeExit() {
        System.out.println("Crawler " + getMyId() + " > Processed: " + stat.getTotalProcessedPages());
    }
}

3. 控制器啟動類 ZiroomController

public class ZiroomController {
    public static void main(String[] args) throws Exception {
        String crawlStorage = "data/crawl/root";
        int numberOfCrawlers = 7;

        CrawlConfig config = new CrawlConfig();
        config.setCrawlStorageFolder(crawlStorage);
        config.setPolitenessDelay(1000); // 禮貌延遲 1s
        config.setMaxPagesToFetch(50);   // 限制抓取頁數

        PageFetcher pageFetcher = new PageFetcher(config);
        RobotstxtConfig robotConfig = new RobotstxtConfig();
        RobotstxtServer robotServer = new RobotstxtServer(robotConfig, pageFetcher);

        CrawlController controller = new CrawlController(config, pageFetcher, robotServer);
        controller.addSeed("https://sh.ziroom.com/z/nl/");

        // 異步啟動，阻塞直到完成
        controller.start(ZiroomCrawler.class, numberOfCrawlers);

        // 聚合統計
        List<Object> localData = controller.getCrawlersLocalData();
        long totalLinks = 0, totalText = 0;
        int totalPages = 0;
        for (Object o : localData) {
            CrawlStat s = (CrawlStat) o;
            totalLinks += s.getTotalLinks();
            totalText += s.getTotalTextSize();
            totalPages += s.getTotalProcessedPages();
        }
        System.out.println("聚合統計：頁面=" + totalPages + "，鏈接=" + totalLinks + "，文本大小=" + totalText);
    }
}

關鍵配置說明

運行結果預覽

-----------爬取路徑：https://sh.ziroom.com/z/nl/
圖片：https://img.ziroom.com/xxx.jpg
地址：自如公寓·某小區,朝陽區,近地鐵10號線
說明：精裝修 拎包入住 獨立衛浴
價格：¥4500/月
...

進階優化方向

1. 分佈式爬蟲：整合 Redis + Kafka 實現 URL 去重與任務分發。
2. 反爬應對：隨機 User-Agent、代理 IP 池、驗證碼識別。
3. 增量爬取：利用 Last-Modified / ETag 或內容指紋判斷更新。
4. 數據持久化：替換 CSV 為 MySQL / Elasticsearch / MongoDB。

項目開源地址

https://github.com/yasserg/crawler4j

總結

crawler4j 以極簡 API 實現了多線程、禮貌爬取、斷點續爬等核心功能，配合 Jsoup 強大的選擇器解析，極大降低了 Java 爬蟲開發門檻。本示例展示了從依賴引入、爬蟲邏輯編寫到數據導出的完整閉環，可作為企業級爬蟲項目的起步模板。