本文介紹使用 Java 實現高併發網頁資料採集的核心技術棧：Jsoup 解析、代理 IP 池管理、反爬蟲應對策略、分散式爬蟲架構設計，幫助開發者構建穩健、高效、合規的資料採集系統。
| 框架/庫 | 優點 | 缺點 | 適用場景 |
|---|---|---|---|
| Jsoup | API 簡潔、支援 CSS 選擇器、容錯性強、無瀏覽器開銷 | 無法執行 JavaScript、無法處理動態渲染頁面 | 靜態頁面、API 介面、結構化資料提取 |
| HttpClient / OkHttp | 底層 HTTP 客戶端、連線池、支援 HTTP/2、協程 | 需自行處理 HTML 解析 | 高併發請求、大檔案下載、微服務通訊 |
| Selenium / Playwright | 真實瀏覽器環境、支援 JS 渲染、模擬用戶行為 | 資源消耗大、啟動慢、難以大規模併發 | 動態渲染、複雜驗證碼、E2E 測試 |
| WebMagic | 國產爬蟲框架、內建調度/去重/持久化、註解式配置 | 社區活躍度一般、擴充性受限 | 快速構建中小型爬蟲專案 |
推薦組合：OkHttp (連線池/HTTP/2) + Jsoup (解析) + Redis (去重/隊列) + 代理池服務
@Data
@Builder
public class ProxyIp {
private String host;
private int port;
private String username; // 私密代理認證
private String password;
private ProxyType type; // HTTP/HTTPS/SOCKS5
private long responseTime; // 驗證延遲 ms
private int successCount;
private int failCount;
private LocalDateTime lastVerified;
private boolean available;
public String toHttpProxyString() {
return (username != null ? username + ":" + password + "@" : "") + host + ":" + port;
}
}
public enum ProxyType { HTTP, HTTPS, SOCKS5 }
@Service
@Slf4j
public class ProxyPoolService {
private final RedisTemplate redisTemplate;
private final OkHttpClient httpClient;
private final Set sourceApis = Set.of(
"https://api.xicidaili.com/free2016.txt",
"https://api.proxyfly.io/proxies?format=txt",
"自建代理服務 API"
);
// 定時任務：每 5 分鐘刷新
@Scheduled(fixedDelay = 300_000)
public void refreshProxyPool() {
List candidates = new ArrayList<>();
for (String api : sourceApis) {
candidates.addAll(fetchFromApi(api));
}
// 並行驗證
List verified = candidates.parallelStream()
.map(this::verifyProxy)
.filter(p -> p.isAvailable())
.collect(Collectors.toList());
// 存入 Redis 有序集合 (分數 = 成功率 * 1000 - responseTime)
for (ProxyIp proxy : verified) {
double score = proxy.getSuccessCount() * 1000.0 / (proxy.getSuccessCount() + proxy.getFailCount() + 1) - proxy.getResponseTime();
redisTemplate.opsForZSet().add("proxy:pool", proxy.toHttpProxyString(), score);
}
// 移除分數過低/過期代理
redisTemplate.opsForZSet().removeRangeByScore("proxy:pool", 0, 100);
log.info("代理池刷新完成，可用: {}", verified.size());
}
private ProxyIp verifyProxy(ProxyIp proxy) {
String testUrl = "https://httpbin.org/ip"; // 返回呼叫端 IP
Request request = new Request.Builder()
.url(testUrl)
.proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxy.getHost(), proxy.getPort())))
.timeout(5, TimeUnit.SECONDS)
.build();
long start = System.currentTimeMillis();
try (Response response = httpClient.newCall(request).execute()) {
if (response.isSuccessful()) {
proxy.setResponseTime(System.currentTimeMillis() - start);
proxy.setSuccessCount(proxy.getSuccessCount() + 1);
proxy.setAvailable(true);
proxy.setLastVerified(LocalDateTime.now());
// 可選：解析響應驗證 IP 是否為代理 IP
} else {
proxy.setFailCount(proxy.getFailCount() + 1);
proxy.setAvailable(false);
}
} catch (Exception e) {
proxy.setFailCount(proxy.getFailCount() + 1);
proxy.setAvailable(false);
}
return proxy;
}
// 獲取高分代理 (支援加權隨機)
public ProxyIp acquireBestProxy() {
Set> top = redisTemplate.opsForZSet()
.reverseRangeWithScores("proxy:pool", 0, 49); // 取前 50
if (top == null || top.isEmpty()) return null;
// 加權隨機選取
List proxies = top.stream().map(ZSetOperations.TypedTuple::getValue).collect(Collectors.toList());
return ProxyIp.parse(proxies.get(ThreadLocalRandom.current().nextInt(proxies.size())));
}
}
@Service
@Slf4j
public class CrawlerExecutor {
private final OkHttpClient client = new OkHttpClient.Builder()
.connectionPool(new ConnectionPool(200, 5, TimeUnit.MINUTES))
.retryOnConnectionFailure(true)
.connectTimeout(10, TimeUnit.SECONDS)
.readTimeout(30, TimeUnit.SECONDS)
.build();
private final ProxyPoolService proxyPool;
private final RedisTemplate redisTemplate;
// 併發控制：信號量限制同時進行的請求數
private final Semaphore semaphore = new Semaphore(100);
public CompletableFuture fetch(String url, CrawlContext ctx) {
return CompletableFuture.supplyAsync(() -> {
semaphore.acquireUninterruptibly();
try {
return doFetchWithRetry(url, ctx, 3);
} finally {
semaphore.release();
}
}, crawlerExecutor); // 自定義執行緒池
}
private Document doFetchWithRetry(String url, CrawlContext ctx, int retry) {
ProxyIp proxy = proxyPool.acquireBestProxy();
Request.Builder builder = new Request.Builder()
.url(url)
.header("User-Agent", rotateUserAgent())
.header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
.header("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8");
if (proxy != null) {
builder.proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxy.getHost(), proxy.getPort())));
if (proxy.getUsername() != null) {
String credential = Credentials.basic(proxy.getUsername(), proxy.getPassword());
builder.header("Proxy-Authorization", credential);
}
}
Request request = builder.build();
long start = System.currentTimeMillis();
try (Response response = client.newCall(request).execute()) {
long cost = System.currentTimeMillis() - start;
if (response.isSuccessful()) {
String html = response.body().string();
// 記錄成功指標
recordMetric(url, cost, true, proxy);
return Jsoup.parse(html, url);
} else {
// 非 2xx：記錄失敗、標記代理降權
recordMetric(url, cost, false, proxy);
if (proxy != null) proxyPool.reportFailure(proxy);
if (retry > 0 && isRetryable(response.code())) {
return doFetchWithRetry(url, ctx, retry - 1);
}
throw new CrawlException("HTTP " + response.code() + ": " + url);
}
} catch (IOException e) {
recordMetric(url, System.currentTimeMillis() - start, false, proxy);
if (proxy != null) proxyPool.reportFailure(proxy);
if (retry > 0) {
sleepWithJitter(1000); // 指數退避 + 抖動
return doFetchWithRetry(url, ctx, retry - 1);
}
throw new CrawlException("Fetch failed: " + url, e);
}
}
private void recordMetric(String url, long cost, boolean success, ProxyIp proxy) {
// 寫入 Redis/InfluxDB 監控
}
}
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Scheduler │────▶│ Redis Queue│────▶│ Workers × N│
│ (種子/去重) │ │ (優先級隊列)│ │ (無狀態) │
└─────────────┘ └─────────────┘ └─────────────┘
│ │
┌──────┴──────┐ ┌──────┴──────┐
│ Proxy Pool │ │ Storage │
│ (服務化) │ │ (ES/MySQL) │
└─────────────┘ └─────────────┘
robots.txt 協議，檢查 User-agent: * 規則構建生產級爬蟲系統的核心在於：代理池高可用、反爬對抗動態化、架構無狀態水平擴展、全鏈路可觀測。Jsoup + OkHttp + Redis + 代理服務化是經典且穩健的技術選型。切記：技術手段服務於合規業務，工程師應恪守法律底線與道德準則。
