文本分析、日誌挖掘、搜索引擎倒排索引構建等場景常需統計單詞出現頻率。本文提供一個健壯、可擴展的實現方案，涵蓋分詞清洗、大小寫歸一化、停用詞過濾、結果排序輸出。
public class WordFrequencyCounter {
private static final Pattern WORD_PATTERN = Pattern.compile("[a-zA-Z]+"); // 僅匹配字母單詞
private static final Set<String> STOP_WORDS = Set.of("the", "and", "or", "a", "an", "in", "on", "to", "of", "for", "with", "is", "are", "was", "were");
public static Map<String, Long> count(String text) {
if (text == null || text.isBlank()) return Collections.emptyMap();
return WORD_PATTERN.matcher(text.toLowerCase(Locale.ROOT))
.results()
.map(MatchResult::group)
.filter(w -> w.length() > 1 && !STOP_WORDS.contains(w)) // 過濾單字母與停用詞
.collect(Collectors.groupingBy(Function.identity(), Collectors.counting()));
}
public static void printTopN(Map<String, Long> freqMap, int n) {
freqMap.entrySet().stream()
.sorted(Map.Entry.<String, Long>comparingByValue().reversed()
.thenComparing(Map.Entry.comparingByKey()))
.limit(n)
.forEach(e -> System.out.printf("%-15s %5d%n", e.getKey(), e.getValue()));
}
public static void main(String[] args) {
String text = "Welcome welcome to ADempiere, a commons-based peer-production of Open Source ERP Applications. " +
"This Wiki is for the global community to contribute and share know-how and domain expertise. " +
"We hope you can find as much open information and participate in making it most usable for everyone.";
Map<String, Long> freq = count(text);
printTopN(freq, 20);
}
}
| 環節 | 方案 | 優勢 |
|---|---|---|
| 分詞 | Pattern.compile("[a-zA-Z]+") | 避免標點、數字干擾，比 split("\\W+") 更精準 |
| 大小寫 | toLowerCase(Locale.ROOT) | 跨平台一致，避免土耳其語 I/ı 問題 |
| 計數 | Stream + Collectors.groupingBy/counting | 無鎖並行友好，代碼簡潔 |
| 停用詞 | Set.of(...) 過濾 | O(1) 查找，可外部化配置 |
| 排序 | 頻率降序 + 字母升序 | 結果穩定可預測 |
WORD_PATTERN 為分詞器迭代器。Files.lines(path) + flatMap 逐行統計，避免 OOM。flatMap -> map -> reduceByKey 模式。單詞頻率統計看似簡單，實則涉及分詞規則、編碼規範、數據結構選型等工程細節。採用「正則預編譯 + Stream 流式聚合 + 可配置停用詞」模式，兼顧性能與可維護性，可直接應用於生產環境文本分析管道。
