Hibernate 實戰：構建網頁爬蟲持久化存儲系統

項目背景與架構

本文演示如何結合 Hibernate ORM 框架，將爬蟲抓取的網頁數據持久化到 MySQL 數據庫。項目結構包含：配置文件、實體類、映射文件、爬蟲核心邏輯、測試入口。

1. 核心配置：hibernate.cfg.xml

<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
    "-//Hibernate/Hibernate Configuration DTD 3.0//EN"
    "https://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
    <session-factory>
        <!-- 數據庫連接配置 -->
        <property name="connection.driver_class">com.mysql.jdbc.Driver</property>
        <property name="connection.url">jdbc:mysql://localhost:3306/hSpider</property>
        <property name="connection.username">root</property>
        <property name="connection.password"></property>
        
        <!-- 方言與優化 -->
        <property name="dialect">org.hibernate.dialect.MySQLDialect</property>
        <property name="cache.provider_class">org.hibernate.cache.NoCacheProvider</property>
        <property name="show_sql">true</property>
        <!-- <property name="hbm2ddl.auto">update</property> -->
        
        <!-- 映射文件引用 -->
        <mapping resource="hibernateSpider/edNews.hbm.xml"/>
    </session-factory>
</hibernate-configuration>

2. 實體類與映射

持久化實體：edNews

package hibernateSpider;

public class edNews {
    private int id;
    private String ednews; // 存儲新聞標題

    public int getId() { return id; }
    public void setId(int id) { this.id = id; }
    public String getNews() { return ednews; }
    public void setNews(String ednews) { this.ednews = ednews; }
}

映射文件：edNews.hbm.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-mapping PUBLIC
    "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
    "https://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping>
    <class name="hibernateSpider.edNews" table="ednews">
        <id name="id" type="int">
            <column name="id" />
            <generator class="increment" />
        </id>
        <property name="news" type="string">
            <column name="news" length="255" />
        </property>
    </class>
</hibernate-mapping>

3. 爬蟲核心實現：Spider.java

package hibernateSpider;

import java.io.*;
import java.net.*;
import java.util.*;
import java.util.regex.*;

public class Spider {
    /** 發送 GET 請求獲取頁面源碼 */
    public static String SendGet(String url) {
        StringBuilder result = new StringBuilder();
        try (BufferedReader in = new BufferedReader(
                new InputStreamReader(new URL(url).openConnection().getInputStream(), "UTF-8"))) {
            String line;
            while ((line = in.readLine()) != null) result.append(line);
        } catch (Exception e) {
            System.err.println("GET 請求異常：" + e.getMessage());
            e.printStackTrace();
        }
        return result.toString();
    }

    /** 正則解析新聞標題 */
    public static ArrayList<News> GetNews(String content) {
        ArrayList<News> results = new ArrayList<>();
        // 匹配類似 ggtz/2017.shtml>標題< 的模式
        Pattern pattern = Pattern.compile("ggtz/\\d{4}\\.shtml.+?>(.+?)<");
        Matcher matcher = pattern.matcher(content);
        
        while (matcher.find()) {
            News news = new News();
            news.ednews = matcher.group(1);
            results.add(news);
        }
        return results;
    }
}

4. 輔助類：News.java（僅用於臨時承載解析數據）

package hibernateSpider;

public class News {
    public String ednews;
    public News() { ednews = ""; }
    @Override public String toString() { return "公告：" + ednews + "\n"; }
}

5. 整合測試：MainTest.java

public class MainTest {
    public static void main(String[] args) {
        String url = "https://jwc.gdut.edu.cn/";
        String content = Spider.SendGet(url);
        ArrayList<News> newsList = Spider.GetNews(content);
        
        // 初始化 Hibernate SessionFactory（建議單例模式，此處為演示簡化）
        Configuration cfg = new Configuration().configure();
        SessionFactory sf = cfg.buildSessionFactory();
        
        for (int i = 0; i < newsList.size(); i++) {
            News n = newsList.get(i);
            System.out.println(n);
            
            // 持久化存儲
            Session session = sf.openSession();
            Transaction tx = null;
            try {
                tx = session.beginTransaction();
                edNews entity = new edNews();
                entity.setId(i);
                entity.setNews(n.ednews);
                session.save(entity);
                tx.commit();
            } catch (Exception e) {
                if (tx != null) tx.rollback();
                e.printStackTrace();
            } finally {
                session.close();
            }
        }
        sf.close();
    }
}

運行結果與優化建議

關鍵優化點：

• SessionFactory 應作為單例全局共享，避免重複構建開銷。
• 批量插入時可使用 JDBC 批處理或 Hibernate 批量操作優化性能。
• 正則表達式需根據目標站點結構動態調整，建議使用 Jsoup 等專業解析庫替代。
• 生產環境需配置連接池（如 C3P0、HikariCP）替代內置簡易池。