java - 运行 Hadoop : insufficient memory for the Java Runtime Environment to continue

Question

我有一个在 3gb 内存上运行的 32 位 linux 系统。当我尝试运行 hadoop 示例时，它失败了，说没有足够的内存分配给 jre。生成的结果是:

hadoop jar mapreduce/hadoop-mapreduce-examples-*.jar grep input output ‘dfs[a-z.]+’
15/01/11 10:17:04 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032
15/01/11 10:17:05 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
15/01/11 10:17:05 INFO input.FileInputFormat: Total input paths to process : 7
15/01/11 10:17:06 INFO mapreduce.JobSubmitter: number of splits:7
15/01/11 10:17:06 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name

15/01/11 10:17:06 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
15/01/11 10:17:06 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
15/01/11 10:17:06 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
15/01/11 10:17:06 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
15/01/11 10:17:06 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
15/01/11 10:17:06 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
15/01/11 10:17:06 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
15/01/11 10:17:06 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
15/01/11 10:17:06 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/01/11 10:17:06 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
15/01/11 10:17:06 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
15/01/11 10:17:07 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1420951126090_0001
15/01/11 10:17:07 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
15/01/11 10:17:07 INFO impl.YarnClientImpl: Submitted application application_1420951126090_0001 to ResourceManager at /127.0.0.1:8032
15/01/11 10:17:07 INFO mapreduce.Job: The url to track the job: http://:8088/proxy/application_1420951126090_0001/
15/01/11 10:17:07 INFO mapreduce.Job: Running job: job_1420951126090_0001
15/01/11 10:17:16 INFO mapreduce.Job: Job job_1420951126090_0001 running in uber mode : false
15/01/11 10:17:16 INFO mapreduce.Job:  map 0% reduce 0%
Java HotSpot(TM) Server VM warning: INFO: os::commit_memory(0xa7b5d000, 32768, 1) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 32768 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/local/hadoop-2.2.0/share/hadoop/hs_err_pid5496.log
#
# Compiler replay data is saved as:
# /usr/local/hadoop-2.2.0/share/hadoop/replay_pid5496.log

是因为我的硬件配置还是hadoop的一些设置错误？

Answer 1

这表示您的虚拟内存已用完，请尝试增加交换空间，或减少堆以将程序的其余部分留给虚拟内存。一个 32 位程序被限制为 ~3 GB 的虚拟内存总量，所以如果它全部分配给堆，这不会为程序运行留下太多空间。相比之下，一个 64 位程序被限制为 128 TB 256 TB，具体取决于操作系统。

顺便说一句，在 Windows 上，一个 32 位程序被限制为大约 1.5 GB 的虚拟内存。

由于 hadoop 是一种大数据解决方案，它通常在更大的机器上运行。例如256 GB 到 1 TB 并不罕见。考虑到现在 32 GB 非常便宜，我会考虑至少买这么多，或者更多的内存。