HttpClient + Jsoup 爬蟲實戰：以網易貴金屬資訊為例

HttpClient + Jsoup 爬蟲實戰：以網易貴金屬資訊為例

技術棧

• HttpClient 4.5+：HTTP 協議客戶端、支援連接池、Cookie 管理、重試機制
• Jsoup 1.14+：HTML 解析器、jQuery 風格選擇器、DOM 操作、反爬蟲應對

目標頁面結構分析

目標：https://fa.163.com/zx/gjs/1/ (網易貴金屬資訊列表)

• 列表容器：.g-news
• 單條新聞：dl 標籤
• 標題/鏈接：dl > a
• 發布時間：span.f-fr
• 摘要：span.f-fl.f-ofe.u-digest

HttpClient 獲取 HTML 流程

public class StockUtils {
    public static String getHtmlByUrl(String url) throws IOException {
        try (CloseableHttpClient client = HttpClients.createDefault()) {
            HttpGet get = new HttpGet(url);
            HttpResponse resp = client.execute(get);
            if (resp.getStatusLine().getStatusCode() == HttpStatus.SC_OK) {
                HttpEntity entity = resp.getEntity();
                return entity != null ? EntityUtils.toString(entity) : null;
            }
        } catch (Exception e) {
            System.err.println("訪問異常: " + url);
            e.printStackTrace();
        }
        return null;
    }
}

Jsoup 解析提取數據

public class StockTest {
    public static void main(String[] args) throws IOException {
        String html = StockUtils.getHtmlByUrl("https://fa.163.com/zx/gjs/1/");
        parserHtml(html);
    }

    public static void parserHtml(String html) {
        Document doc = Jsoup.parse(html);
        Elements items = doc.getElementsByClass("g-news").select("dl");
        
        for (Element item : items) {
            String title = item.select("a").text();
            String link = item.select("a").attr("href");
            String time = item.select("span.f-fr").text();
            String summary = item.select("span.f-fl.f-ofe.u-digest").text();
            
            System.out.println("標題: " + title);
            System.out.println("鏈接: " + link);
            System.out.println("時間: " + time);
            System.out.println("摘要: " + summary);
            System.out.println("=================================");
        }
    }
}

運行輸出

進階優化建議

• 連接池複用：單例 CloseableHttpClient、配置 PoolingHttpClientConnectionManager
• 反爬應對：隨機 User-Agent、請求間隔、代理 IP 池、Cookie 持久化
• 異步並發：HttpAsyncClient 或 CompletableFuture + 線程池
• 增量爬取：記錄最後爬取時間/ID、配合資料庫去重
• 結構化存儲：直接映射到 Entity、入庫 MySQL/ES

依賴坐標

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.14</version>
</dependency>
<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.15.4</version>
</dependency>

源碼下載：百度網盤 | 作者博客：cnblogs.com/clayzhang