java - CDH5.2 : MR, 无法初始化任何输出收集器

Question

Cloudera CDH5.2 快速启动虚拟机Cloudera Manager 显示所有节点状态 = 绿色

我已经在 Eclipse 上创建了一个 MR 作业，包括构建路径中的所有相关的 cloudera jar:avro-1.7.6-cdh5.2.0.jar,avro-mapred-1.7.6-cdh5.2.0-hadoop2.jar,hadoop-common-2.5.0-cdh5.2.0.jar,hadoop-mapreduce-client-core-2.5.0-cdh5.2.0.jar

我已经运行了以下作业

hadoop jar jproject1.jar avro00.AvroUserPrefCount -libjars ${LIBJARS} avro/00/in avro/00/out

我收到以下错误，是否是 Java 堆问题，有什么意见吗？提前谢谢你

14/11/14 01:02:40 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
14/11/14 01:02:43 INFO input.FileInputFormat: Total input paths to process : 1
14/11/14 01:02:43 INFO mapreduce.JobSubmitter: number of splits:1
14/11/14 01:02:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1415950730849_0001
14/11/14 01:02:45 INFO impl.YarnClientImpl: Submitted application application_1415950730849_0001
14/11/14 01:02:45 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1415950730849_0001/
14/11/14 01:02:45 INFO mapreduce.Job: Running job: job_1415950730849_0001
14/11/14 01:03:04 INFO mapreduce.Job: Job job_1415950730849_0001 running in uber mode : false
14/11/14 01:03:04 INFO mapreduce.Job:  map 0% reduce 0%
14/11/14 01:03:11 INFO mapreduce.Job: Task Id : attempt_1415950730849_0001_m_000000_0, Status : FAILED
Error: java.io.IOException: Unable to initialize any output collector
    at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:412)
    at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
    at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
...
...

Answer 1

检查失败尝试的完整任务日志 attempt_1415950730849_0001_m_000000_0 将有助于了解您遇到给定异常的原因。

观察到此类错误的最常见原因是作业中 io.sort.mb 的值配置错误。它的值绝不能接近(或高于)配置的映射任务堆大小，并且当前也不得超过 ~2000 MB(Java 数组最大大小)。

最近通过MAPREDUCE-6194 也提交并解决了上游改进，使错误更清楚地表明真正的失败。 .