java - 运行子错误 : java. lang.OutOfMemoryError: Java 堆空间

Question

我在互联网上看了很多，但没有找到解决我问题的方法。我使用 Hadoop 2.6.0。

MapReduce 的主要目标是运行 SequenceFile 并对键/值对进行一些分析。

2015-01-29 10:09:50,554 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output
2015-01-29 10:09:50,554 INFO [main] org.apache.hadoop.mapred.MapTask: Spilling map output
2015-01-29 10:09:50,554 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 23342; bufvoid = 104857600
2015-01-29 10:09:50,554 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 26213840(104855360); length = 557/6553600
2015-01-29 10:09:50,570 INFO [main] org.apache.hadoop.mapred.MapTask: Finished spill 0
2015-01-29 10:09:50,577 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:66)
    at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)
    at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2359)
    at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2491)
    at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:72)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

这里是STDOUT的输出

15/01/29 10:09:35 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
15/01/29 10:09:35 INFO compress.CodecPool: Got brand-new compressor [.gz]

15/01/29 10:09:36 INFO client.RMProxy: Connecting to ResourceManager at xxxxxxxxxxxxxxxxxxxxx:8040
15/01/29 10:09:37 INFO input.FileInputFormat: Total input paths to process : 1
15/01/29 10:09:37 INFO mapreduce.JobSubmitter: number of splits:1
15/01/29 10:09:37 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1422374835659_0059
15/01/29 10:09:37 INFO impl.YarnClientImpl: Submitted application application_1422374835659_0059
15/01/29 10:09:37 INFO mapreduce.Job: The url to track the job: http://xxxxxxxxxxxxxxxxxxxxx:8088/proxy/application_1422374835659_0059/
15/01/29 10:09:37 INFO mapreduce.Job: Running job: job_1422374835659_0059
15/01/29 10:09:44 INFO mapreduce.Job: Job job_1422374835659_0059 running in uber mode : false
15/01/29 10:09:44 INFO mapreduce.Job:  map 0% reduce 0%
15/01/29 10:09:50 INFO mapreduce.Job: Task Id : attempt_1422374835659_0059_m_000000_0, Status : FAILED
Error: Java heap space
15/01/29 10:09:58 INFO mapreduce.Job: Task Id : attempt_1422374835659_0059_m_000000_1, Status : FAILED
Error: Java heap space
15/01/29 10:10:04 INFO mapreduce.Job: Task Id : attempt_1422374835659_0059_m_000000_2, Status : FAILED
Error: Java heap space
15/01/29 10:10:10 INFO mapreduce.Job:  map 100% reduce 100%
15/01/29 10:10:11 INFO mapreduce.Job: Job job_1422374835659_0059 failed with state FAILED due to: Task failed task_1422374835659_0059_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

15/01/29 10:10:11 INFO mapreduce.Job: Counters: 12
    Job Counters 
        Failed map tasks=4
        Launched map tasks=4
        Other local map tasks=3
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=37910
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=18955
        Total vcore-seconds taken by all map tasks=18955
        Total megabyte-seconds taken by all map tasks=38819840
    Map-Reduce Framework
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0

我的配置几乎是默认的，与 Java 堆大小无关。

我也试过这个，没有什么区别。

<property>
        <name>mapred.child.java.opts</name>
        <value>-Xmx1024m</value>
</property>

master 上的主程序以 -Xmx512m 开头，而节点上的容器以 -Xmx1024m 开头 符合预期。

同时编辑 hadoop-env.sh对以下内容没有帮助:

export HADOOP_CLIENT_OPTS="-Xmx2048m $HADOOP_CLIENT_OPTS"

我的 MapReduce 应用程序中的配置:

conf.setInt("mapreduce.map.memory.mb", 2048);
conf.setInt("mapreduce.reduce.memory.mb", 1024);

编辑 1 29.01:

与 -Xmx2048m我收到了同样的错误。

与 -Xmx3072m我转到以下错误:

Error: java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:197)
    at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:70)
    at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)
    at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2359)
    at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2491)
    at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:72)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

与 -Xmx4096m我遇到了一个完全不同的错误，我不明白为什么他现在要使用 5GB 的虚拟内存:

Container [pid=61687,containerID=container_1422374835659_0064_01_000002] is running beyond virtual memory limits. Current usage: 866.8 MB of 2 GB physical memory used; 5.0 GB of 4.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_1422374835659_0064_01_000002 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 61687 61685 61687 61687 (bash) 0 0 12640256 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx4096m -Djava.io.tmpdir=/home/hduser/tmp/nm-local-dir/usercache/hduser/appcache/application_1422374835659_0064/container_1422374835659_0064_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop-2.6.0/logs/userlogs/application_1422374835659_0064/container_1422374835659_0064_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.97.83.13 33802 attempt_1422374835659_0064_m_000000_0 2 1>/usr/local/hadoop-2.6.0/logs/userlogs/application_1422374835659_0064/container_1422374835659_0064_01_000002/stdout 2>/usr/local/hadoop-2.6.0/logs/userlogs/application_1422374835659_0064/container_1422374835659_0064_01_000002/stderr  
    |- 61692 61687 61687 61687 (java) 629 149 5384613888 221601 /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx4096m -Djava.io.tmpdir=/home/hduser/tmp/nm-local-dir/usercache/hduser/appcache/application_1422374835659_0064/container_1422374835659_0064_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop-2.6.0/logs/userlogs/application_1422374835659_0064/container_1422374835659_0064_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.97.83.13 33802 attempt_1422374835659_0064_m_000000_0 2 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

编辑 2 29.01

即使在 map() 函数中注释掉所有内容，错误也会出现。

SequenceFile (132.93 KB) 中只有 10 个键/值对，一切正常。

编辑 3 30.01

这里是最小化的 Source，它会产生相同的错误。

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class Dummy implements Tool {

    private Configuration conf;

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        int res = ToolRunner.run(conf, new Dummy(), args);
        System.exit(res);
    }

    @Override
    public void setConf(Configuration conf) {
        // Set some Job options
        conf.set("dfs.blocksize", "16m");

        // set heap size
        // conf.set("yarn.app.mapreduce.am.command-opts", "-Xmx1024m");
        // conf.set("mapred.child.java.opts", "-Xmx200m");

        // request more memory be the ressourcemanager
        conf.setInt("mapreduce.map.memory.mb", 2048);
        conf.setInt("mapreduce.reduce.memory.mb", 1024);

        // IO space
        // conf.setInt("mapreduce.task.io.sort.mb", 256);

        // Since we have lots of small tasks we should reduce overhead
        // conf.setInt("mapreduce.job.jvm.numtasks", -1);

        this.conf = conf;
    }

    /**
     * configuration getter
     */
    @Override
    public Configuration getConf() {
        return conf;
    }

    @Override
    public int run(String[] args) throws IOException, ClassNotFoundException, InterruptedException {

        // Configure the job
        Job job = Job.getInstance(conf, "Dummy");

        job.setJarByClass(Dummy.class);

        job.setInputFormatClass(SequenceFileInputFormat.class);
        job.setMapperClass(Map.class);

        // Set number of Reducers to number of actions + 1 for error log
        // job.setNumReduceTasks(action_count+2);
        job.setReducerClass(Reduce.class); // Global Aggregation

        // Set output
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);

        // Enable record skipping for failed Maps
        // SkipBadRecords.setMapperMaxSkipRecords(conf, Long.MAX_VALUE);

        // only create a output file it there is content
        // LazyOutputFormat.setOutputFormatClass(job, TextOutputFormat.class);

        // set input and output for job
        // FileInputFormat.addInputPath(job, repo.getRepository());
        FileInputFormat.setInputPaths(job, new Path("/test/test.seq"));
        FileOutputFormat.setOutputPath(job, new Path("/test/out"));

        // Execute Job
        int res = 0;
        // job.submit();
        res = job.waitForCompletion(true) ? 0 : 1;

        return res;

    }

    public static class Map extends Mapper<Text, Text, Text, Text> {
        @Override
        protected void map(Text key, Text value, Mapper<Text, Text, Text, Text>.Context context) {
            // TODO Auto-generated method stub
        }
    }

    public static class Reduce extends Reducer<Text, Text, Text, Text> {

        @Override
        protected void reduce(Text key, Iterable<Text> value, Reducer<Text, Text, Text, Text>.Context context) {
            // TODO Auto-generated method stub

        }
    }
}

Answer 1

我最近遇到了同样的问题。我正在使用 Oracle VM 来学习 hadoop。分配的基本内存为 512 MB，我遇到了同样的错误:

java.lang.Exception: java.lang.OutOfMemoryError: Java heap space

然后我将它增加到 1024MB，然后我能够成功运行 MR 程序。