Java 爬蟲實戰：基於 Jsoup 采集 CSDN 博客文章

專案結構與依賴

所需 JAR：jsoup-1.10.2.jar

實體類：Article

public class Article {
    private String address; // 相對鏈接
    private String title;   // 標題
    private String desption; // 簡介
    private String time;    // 發表時間
    // Getter/Setter 省略
}

單頁采集器：First

public class First {
    private static final String URL = "https://blog.csdn.net/qq_33599520/article/list/1";

    public static void main(String[] args) throws IOException {
        Connection conn = Jsoup.connect(URL)
            .userAgent("Mozilla/5.0 (Windows NT 6.1; rv:47.0) Gecko/20100101 Firefox/47.0")
            .timeout(5000)
            .method(Connection.Method.GET);

        Document doc = conn.get();
        Element body = doc.body();

        List<Article> resultList = new ArrayList<>();
        Element articleListDiv = body.getElementById("article_list");
        Elements articleList = articleListDiv.getElementsByClass("list_item");

        for (Element article : articleList) {
            Article entity = new Article();
            Element linkNode = article.select("div h1 a").get(0);
            Element desptionNode = article.getElementsByClass("article_description").get(0);
            Element manageNode = article.getElementsByClass("article_manage").get(0);

            entity.setAddress(linkNode.attr("href"));
            entity.setTitle(linkNode.text());
            entity.setDesption(desptionNode.text());
            entity.setTime(manageNode.select("span:eq(0)").text()); // 注意選擇器語法
            resultList.add(entity);
        }

        System.out.println("文章總數：" + resultList.size());
        for (Article a : resultList) {
            System.out.println("絕對路徑：https://blog.csdn.net" + a.getAddress());
        }
    }
}

全站采集器：Main（支持分頁）

public class Main {
    private static final String URL = "https://blog.csdn.net/qq_33599520";

    public static void main(String[] args) throws IOException {
        // 獲取總頁數
        Connection conn = Jsoup.connect(URL)
            .userAgent("Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0")
            .timeout(5000)
            .method(Connection.Method.GET);
        Document doc = conn.get();
        String totalPageStr = doc.body().getElementById("papelist").select("span:eq(0)").text();
        totalPageStr = totalPageStr.replaceAll(".+共(\\d+)頁", "$1");
        int totalPage = Integer.parseInt(totalPageStr);

        List<Article> allArticles = new ArrayList<>();
        for (int page = 1; page <= totalPage; page++) {
            allArticles.addAll(getArticlesByPage(page));
        }

        // 輸出所有文章
        for (Article a : allArticles) {
            System.out.println("標題：" + a.getTitle());
            System.out.println("鏈接：https://blog.csdn.net" + a.getAddress());
            System.out.println("簡介：" + a.getDesption());
            System.out.println("時間：" + a.getTime());
        }
    }

    private static List<Article> getArticlesByPage(int page) throws IOException {
        // 邏輯同 First 類，針對 /article/list/{page} 頁面解析
        // 代碼省略，結構相同
        return new ArrayList<>();
    }
}

關鍵技術點

• 設置 User-Agent 模擬瀏覽器，防反爬。
• 使用 CSS 選擇器（select、getElementsByClass、getElementById）精準定位元素。
• 分頁邏輯：提取總頁數 → 循環請求每頁 → 合併結果。