Java 爬蟲抓取京東行政區劃數據：Jsoup 解析與多層級遞迴完整實作

專案背景

從京東收貨地址介面抓取省、市、區、鎮四級行政區劃，輸出 JSON 供前端級聯選單使用。

核心介面

• 省份：`https://trade.jd.com/dynamic/consignee/getProvinces.action`
• 城市：`https://trade.jd.com/dynamic/consignee/getCitys.action?consigneeParam.provinceId=`
• 區縣：`https://trade.jd.com/dynamic/consignee/getCountys.action?consigneeParam.cityId=`
• 鄉鎮：`https://trade.jd.com/dynamic/consignee/getTowns.action?consigneeParam.countyId=`

資料模型

class Address {
    private String id;
    private String name;
    private List
 children;
    // getter/setter 省略
}

完整爬蟲代碼

package com.test.html;

import com.alibaba.fastjson.JSON;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang.StringUtils;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

import java.io.File;
import java.io.IOException;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;

public class JD_Address {
    private static final int TIMEOUT = 300000;
    private static int count = 0;

    public static void main(String[] args) throws IOException, InterruptedException {
        String provinceUrl = "https://trade.jd.com/dynamic/consignee/getProvinces.action";
        String cityUrl = "https://trade.jd.com/dynamic/consignee/getCitys.action?consigneeParam.provinceId=";
        String countyUrl = "https://trade.jd.com/dynamic/consignee/getCountys.action?consigneeParam.cityId=";
        String townUrl = "https://trade.jd.com/dynamic/consignee/getTowns.action?consigneeParam.countyId=";

        List
 provinces = getList(provinceUrl);
        for (Address province : provinces) {
            List
 cities = getList(cityUrl + province.getId());
            province.setChildren(cities);
            for (Address city : cities) {
                List
 counties = getList(countyUrl + city.getId());
                city.setChildren(counties);
                for (Address county : counties) {
                    List
 towns = getList(townUrl + county.getId());
                    county.setChildren(towns);
                }
            }
        }

        FileUtils.writeStringToFile(new File("address.js"), "var ds=" + JSON.toJSONString(provinces), "UTF-8");
        System.out.println("完成，共 " + count + " 次請求");
    }

    private static List
 getList(String url) throws IOException, InterruptedException {
        List
 list = new ArrayList<>();
        Document doc = Jsoup.parse(new URL(url), TIMEOUT);
        Elements options = doc.select("option[value~=\\d+]");
        for (Element el : options) {
            Address addr = new Address();
            addr.setId(el.attr("value"));
            addr.setName(el.text().replace("*", ""));
            list.add(addr);
        }
        count++;
        if (count % 100 == 0) System.out.println("已請求: " + count);
        if (count % 500 == 0) Thread.sleep(5000); // 禮貌性延遲
        return list;
    }
}

輸出 JSON 結構範例

var ds = [{
  "id": "1", "name": "北京",
  "children": [{
    "id": "72", "name": "朝陽區",
    "children": [{
      "id": "2799", "name": "三環之內", "children": []
    }, ...]
  }, ...]
}, {
  "id": "2", "name": "上海", "children": [...] }
];

關鍵技術點

• Jsoup 解析 HTML：`select("option[value~=\\d+]")` 精準提取下拉選項。
• 遞迴四層：省→市→區→鎮，層層組裝樹狀結構。
• Fastjson 序列化：直接輸出 JS 變數賦值語句，前端直引用。
• 反爬應對：設定超時、每 500 請求睡眠 5 秒、User-Agent 模擬（Jsoup 預設即可）。

依賴座標

    org.jsoup
    jsoup
    1.16.1


    com.alibaba
    fastjson
    1.2.83


    commons-io
    commons-io
    2.11.0


    org.apache.commons
    commons-lang3
    3.12.0