Java 爬蟲入門：HttpURLConnection 抓取網頁內容完整實現

兩種核心 API 對比

簡易版：一行獲取字符串

public static String getContent(String urlStr) {
    try (BufferedReader br = new BufferedReader(
            new InputStreamReader(new URL(urlStr).openStream(), "UTF-8"))) {
        return br.lines().collect(Collectors.joining("\n"));
    } catch (Exception e) {
        return "error: " + e.getMessage();
    }
}

生產級版：完整 HttpURLConnection 封裝

public class HttpClientUtil {
    public static String get(String urlStr, String proxyHost, int proxyPort,
                             String username, String password) throws IOException {
        // 1. 代理設置
        if (proxyHost != null) {
            Authenticator.setDefault(new Authenticator() {
                protected PasswordAuthentication getPasswordAuthentication() {
                    return new PasswordAuthentication(username, password.toCharArray());
                }
            });
            System.setProperty("http.proxyHost", proxyHost);
            System.setProperty("http.proxyPort", String.valueOf(proxyPort));
        }

        // 2. 打開連接
        HttpURLConnection conn = (HttpURLConnection) new URL(urlStr).openConnection();
        conn.setConnectTimeout(10_000);
        conn.setReadTimeout(30_000);
        conn.setRequestProperty("User-Agent", "Mozilla/5.0 (compatible; JavaBot)");
        conn.setRequestProperty("Accept", "text/html,application/xhtml+xml");
        conn.connect();

        // 3. 讀取響應
        try (InputStream is = conn.getInputStream();
             BufferedReader br = new BufferedReader(new InputStreamReader(is, "UTF-8"))) {
            return br.lines().collect(Collectors.joining("\n"));
        }
    }
}

關鍵最佳實踐

• 編碼自動檢測：優先用 Content-Type 中的 charset，回退 UTF-8，再用 cpDetector 或 juniversalchardet
• 重定向處理：conn.setInstanceFollowRedirects(true)（默認開啟）
• 壓縮支持：conn.setRequestProperty("Accept-Encoding", "gzip, deflate")，讀取時用 GZIPInputStream 解壓
• 連接池：高併發場景請改用 Apache HttpClient / OkHttp，原生 HttpURLConnection 無連接池
• 異常細分：分別捕獲 SocketTimeoutException、ConnectException、SSLHandshakeException 做重試/告警

總結

標準庫 HttpURLConnection 足夠應對簡單爬蟲、內部服務調用；涉及連接池、連接保活、HTTP/2、異步非阻塞時，請直接引入 OkHttp 或 Apache HttpClient 5，避免造輪子。