java - 我无法在以独立模式配置的 hadoop 上执行 map-reduce 作业

Question

我正在尝试在我的计算机 (MacOS 10.7) 上的本地文件系统(独立模式)上测试一个非常简单的 hadoop map-reduce 作业。该作业采用 .csv 文件 (data-01) 并计算某些字段的出现次数。

我下载了 CDH4 hadoop，运行作业，它似乎正常启动，但在处理完所有拆分后，我收到以下错误:

13/03/12 12:11:18 INFO mapred.MapTask: Processing split: file:/path/in/data-01:9999220736+33554432
13/03/12 12:11:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
13/03/12 12:11:18 INFO mapred.LocalJobRunner: Starting task: attempt_local2133287029_0001_m_000299_0
13/03/12 12:11:18 INFO mapred.Task:  Using ResourceCalculatorPlugin : null
13/03/12 12:11:18 INFO mapred.MapTask: Processing split: file:/path/in/data-01:10032775168+33554432
13/03/12 12:11:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
13/03/12 12:11:18 INFO mapred.LocalJobRunner: Starting task: attempt_local2133287029_0001_m_000300_0
13/03/12 12:11:18 INFO mapred.Task:  Using ResourceCalculatorPlugin : null
13/03/12 12:11:18 INFO mapred.MapTask: Processing split: file:/path/in/data-01:10066329600+33554432
13/03/12 12:11:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
13/03/12 12:11:18 INFO mapred.LocalJobRunner: Starting task: attempt_local2133287029_0001_m_000301_0
13/03/12 12:11:18 INFO mapred.Task:  Using ResourceCalculatorPlugin : null
13/03/12 12:11:18 INFO mapred.MapTask: Processing split: file:/path/in/data-01:10099884032+33554432
13/03/12 12:11:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
13/03/12 12:11:18 INFO mapred.LocalJobRunner: Starting task: attempt_local2133287029_0001_m_000302_0
13/03/12 12:11:18 INFO mapred.Task:  Using ResourceCalculatorPlugin : null
13/03/12 12:11:18 INFO mapred.MapTask: Processing split: file:/path/in/data-01:10133438464+32025555
13/03/12 12:11:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
13/03/12 12:11:19 INFO mapred.LocalJobRunner: Map task executor complete.
13/03/12 12:11:19 WARN mapred.LocalJobRunner: job_local2133287029_0001
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:399)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:949)
    at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:389)
    at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:78)
    at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:668)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:740)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:231)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
13/03/12 12:11:19 INFO mapreduce.Job: Job job_local2133287029_0001 failed with state FAILED due to: NA
13/03/12 12:11:19 INFO mapreduce.Job: Counters: 0

无论输入文件多小，我都会得到同样的错误...

Answer 1

碰巧默认选项取代了我的本地配置(我仍然不明白为什么)。

export HADOOP_CLIENT_OPTS="-Xmx1024m"

解决了问题。