Java 爬蟲批量抓取 豌豆莢 Android 應用圖標與包名

Java 爬蟲實戰：抓取 豌豆莢「全部軟體」排行榜前 20 頁應用圖標與包名

Android 主題開發需替換第三方應用圖標，必須知道應用包名。豌豆莢網頁版應用詳情頁 URL 末段即為包名。本爬蟲示範使用 HttpClient + Jsoup 解析列表頁，批量下載圖標並建立「應用名 ↔ 包名」對照表。

專案依賴

<dependencies>
  <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.5.13</version> </dependency>
  <dependency> <groupId>org.jsoup</groupId> <artifactId>jsoup</artifactId> <version>1.15.4</version> </dependency>
  <dependency> <groupId>commons-io</groupId> <artifactId>commons-io</artifactId> <version>2.11.0</version> </dependency>
</dependencies>

核心爬蟲類

import org.apache.http.*;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.*;
import java.nio.file.*;
import java.util.*;

public class WandoujiaIconCrawler {

    private static final String BASE_URL = "https://www.wandoujia.com/tag/%E5%85%A8%E9%83%A8%E8%BD%AF%E4%BB%B6/total?page=";
    private static final Path ICON_DIR = Paths.get("icons");
    private static final Path MAP_FILE = ICON_DIR.resolve("packageName.txt");

    public static void main(String[] args) throws Exception {
        Files.createDirectories(ICON_DIR);
        try (CloseableHttpClient httpClient = HttpClients.createDefault();
             BufferedWriter mapWriter = Files.newBufferedWriter(MAP_FILE, StandardCharsets.UTF_8)) {

            for (int page = 1; page <= 20; page++) {
                System.out.println("=== 抓取第 " + page + " 頁 ===");
                String url = BASE_URL + page;
                String html = fetchHtml(httpClient, url);
                if (html == null) continue;

                Document doc = Jsoup.parse(html);
                Elements appLinks = doc.select("div.container.clearfix > section.main-col > div.app-blocks > div.app-block > ul.app-list.clearfix > li.app > a.icon-area");
                Elements nameSpans = doc.select("div.container.clearfix > section.main-col > div.app-blocks > div.app-block > ul.app-list.clearfix > li.app > div.operate > a.name > span.txt");

                for (int i = 0; i < appLinks.size(); i++) {
                    Element link = appLinks.get(i);
                    String pkg = link.attr("data-pn"); // 包名
                    Element img = link.selectFirst("img.icon");
                    if (img == null) continue;
                    String imgUrl = img.attr("src").replace("68_68", "256_256"); // 取大圖
                    String name = (i < nameSpans.size()) ? nameSpans.get(i).text() : "Unknown";

                    // 下載圖標
                    String fileName = pkg + ".png";
                    Path dest = ICON_DIR.resolve(fileName);
                    if (downloadImage(httpClient, imgUrl, dest)) {
                        mapWriter.write("【" + name + "】" + pkg);
                        mapWriter.newLine();
                        System.out.println("  ✓ " + name + " (" + pkg + ")");
                    } else {
                        System.err.println("  ✗ 下載失敗: " + imgUrl);
                    }
                    Thread.sleep(50); // 禮貌延遲
                }
            }
        }
        System.out.println("完成！圖標存放於 " + ICON_DIR.toAbsolutePath());
    }

    private static String fetchHtml(CloseableHttpClient client, String url) {
        HttpGet get = new HttpGet(url);
        get.setHeader("User-Agent", "Mozilla/5.0");
        try (CloseableHttpResponse resp = client.execute(get)) {
            if (resp.getStatusLine().getStatusCode() == 200) {
                return EntityUtils.toString(resp.getEntity(), "UTF-8");
            }
        } catch (IOException e) {
            System.err.println("請求失敗: " + url + " - " + e.getMessage());
        }
        return null;
    }

    private static boolean downloadImage(CloseableHttpClient client, String url, Path dest) {
        HttpGet get = new HttpGet(url);
        get.setHeader("User-Agent", "Mozilla/5.0");
        try (CloseableHttpResponse resp = client.execute(get);
             InputStream in = resp.getEntity().getContent();
             OutputStream out = Files.newOutputStream(dest)) {
            if (resp.getStatusLine().getStatusCode() == 200) {
                byte[] buf = new byte[8192];
                int len;
                while ((len = in.read(buf)) != -1) out.write(buf, 0, len);
                return true;
            }
        } catch (IOException e) {
            System.err.println("下載異常: " + e.getMessage());
        }
        return false;
    }
}

關鍵技術點

1. CSS 選擇器：針對列表頁結構精確定位 `` 與圖片 ``。
2. 圖片 URL 轉換：縮圖 URL 含 `68_68`，替換為 `256_256` 取得高清圖標。
3. 資源管理：`try-with-resources` 確保 HTTP 連線、輸入輸出流正確關閉。
4. 禮貌爬蟲：`Thread.sleep(50)` 降低對目標站點壓力，避免被封 IP。
5. 斷點續傳：可增加檢查 `Files.exists(dest)` 跳過已下載檔案。

• Puppeteer/Playwright：無頭瀏覽器，應對動態渲染、反爬更強。
• WebMagic：Java 爬蟲框架，支援分佈式、調度、去重。
• 官方應用商店 API (Google Play Developer API) 獲取權威元數據。