hadoop - MapReduce 作业卡在多节点 Hadoop-2.7.1 集群上

Question

我已在多节点集群(1 个名称节点和 4 个数据节点)上成功运行 Hadoop 2.7.1。但是，当我运行 MapReduce 作业(来自 Hadoop 网站的 WordCount 示例)时，它总是停留在这一点上。

[~@~ hadoop-2.7.1]$ bin/hadoop jar WordCount.jar WordCount /user/inputdata/ /user/outputdata
15/09/30 17:54:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/09/30 17:54:57 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/09/30 17:54:58 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/09/30 17:54:59 INFO input.FileInputFormat: Total input paths to process : 1
15/09/30 17:55:00 INFO mapreduce.JobSubmitter: number of splits:1
15/09/30 17:55:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1443606819488_0002
15/09/30 17:55:00 INFO impl.YarnClientImpl: Submitted application application_1443606819488_0002
15/09/30 17:55:00 INFO mapreduce.Job: The url to track the job: http://~~~~:8088/proxy/application_1443606819488_0002/
15/09/30 17:55:00 INFO mapreduce.Job: Running job: job_1443606819488_0002

我必须为 yarn 指定内存吗？

NOTE: DataNode hardwares are really old (Each has 1GB RAM).

感谢您的帮助。谢谢。

Answer 1

数据节点内存(1gb)确实非常稀缺，无法准备至少 1 个容器在其中运行 mapper/reducer/am。

您可以尝试使用非常低的值降低 yarn-site.xml 中的以下容器内存分配值，以便在其上创建容器。

yarn.scheduler.minimum-allocation-mb
yarn.scheduler.maximum-allocation-mb

同时尝试减少作业配置中的以下属性值，

mapreduce.map.memory.mb
mapreduce.reduce.memory.mb
mapreduce.map.java.opts
mapreduce.reduce.java.opts