Maven 搭建 Hadoop 開發環境：從專案建立到 MapReduce 實戰

一、建立 Maven 專案

mvn archetype:generate \
  -DgroupId=my.hadoopstudy \
  -DartifactId=hadoopstudy \
  -DarchetypeArtifactId=maven-archetype-quickstart \
  -DinteractiveMode=false

二、pom.xml 引入 Hadoop 相依套件

<dependencies>
  <dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>2.5.1</version>
  </dependency>
  <dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>2.5.1</version>
  </dependency>
  <dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.5.1</version>
  </dependency>
  <dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>3.8.1</version>
    <scope>test</scope>
  </dependency>
</dependencies>

提示：版本號請與叢集端 Hadoop 版本一致，避免序列化不相容。

三、HDFS 操作測試

package my.hadoopstudy.dfs;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.io.IOUtils;
import java.io.InputStream;
import java.net.URI;

public class Test {
    public static void main(String[] args) throws Exception {
        String uri = "hdfs://9.111.254.189:9000/";
        Configuration conf = new Configuration();
        FileSystem fs = FileSystem.get(URI.create(uri), conf);

        // 列出目錄
        FileStatus[] statuses = fs.listStatus(new Path("/user/fkong"));
        for (FileStatus s : statuses) System.out.println(s);

        // 建立檔案並寫入
        try (FSDataOutputStream os = fs.create(new Path("/user/fkong/test.log"))) {
            os.write("Hello World!".getBytes());
        }

        // 讀取檔案
        try (InputStream is = fs.open(new Path("/user/fkong/test.log"))) {
            IOUtils.copyBytes(is, System.out, 1024, true);
        }
    }
}

四、MapReduce WordCount 風格範例：統計 Event 類型次數

package my.hadoopstudy.mapreduce;

import org.apache.hadoop.conf.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
import java.io.IOException;

public class EventCount {

    public static class MyMapper extends Mapper<Object, Text, Text, IntWritable> {
        private final static IntWritable ONE = new IntWritable(1);
        private Text event = new Text();

        @Override
        protected void map(Object key, Text value, Context context) throws IOException, InterruptedException {
            int idx = value.toString().indexOf(' ');
            if (idx > 0) {
                event.set(value.toString().substring(0, idx));
                context.write(event, ONE);
            }
        }
    }

    public static class MyReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
        private IntWritable result = new IntWritable();

        @Override
        protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable v : values) sum += v.get();
            result.set(sum);
            context.write(key, result);
        }
    }

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
        if (otherArgs.length < 2) {
            System.err.println("Usage: EventCount <in> <out>");
            System.exit(2);
        }
        Job job = Job.getInstance(conf, "event count");
        job.setJarByClass(EventCount.class);
        job.setMapperClass(MyMapper.class);
        job.setCombinerClass(MyReducer.class);
        job.setReducerClass(MyReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
        FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

五、打包與叢集提交

1. 打包：mvn package → 產生 target/hadoopstudy-1.0-SNAPSHOT.jar。
2. 上傳 JAR：複製到 Hadoop 節點（如 /opt/hadoop/）。
3. 準備輸入資料：

   mkdir -p /tmp/input
   echo -e "JOB_NEW ...\nJOB_NEW ...\nJOB_FINISH ..." > /tmp/input/event.log.1
   cp /tmp/input/event.log.1 /tmp/input/event.log.2
   cp /tmp/input/event.log.1 /tmp/input/event.log.3
   hdfs dfs -put /tmp/input /user/fkong/input

4. 執行 Job：

   hadoop jar hadoopstudy-1.0-SNAPSHOT.jar my.hadoopstudy.mapreduce.EventCount \
     /user/fkong/input /user/fkong/output

5. 查看結果：

   hdfs dfs -cat /user/fkong/output/part-r-00000
   # 輸出範例：
   # JOB_FINISH    2
   # JOB_NEW       4

六、常見坑點

• 版本不匹配：編譯用 2.5.1，叢集跑 3.x 會報 Unsupported major.minor version 或序列化錯誤。
• 缺少 Hadoop 設定檔：本機執行需將叢集 core-site.xml、hdfs-site.xml、mapred-site.xml 放在 src/main/resources 或指定 HADOOP_CONF_DIR。
• Windows 本機測試：需 winutils.exe 並設定 hadoop.home.dir，建議直接在 Linux/Mac 或 Docker 開發。

總結

Maven 管理 Hadoop 相依極為簡潔，核心在於版本對齊與設定檔同步。掌握 FileSystem API 與 MapReduce Mapper/Reducer 撰寫流程，即可在 IDE 開發、打包、提交叢集完整跑通大數據作業。