hadoop - InvalidResourceRequestException:执行Hadoop Streaming MR作业时出现奇怪错误

Question

执行以下命令时出现此奇怪错误:

hadoop jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.5.1.jar  -file ./methratio.py -file './genome.fa' -mapper './methratio.py -r -g ' -input  /TextLab/sravisha_test/SamFiles/test_bam  -output ./outfile

错误 :

    15/01/28 18:56:29 ERROR streaming.StreamJob: Error Launching job : org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=3072maxMemory=2583 at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtil s.validateResourceRequest(SchedulerUtils.java:196)  at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateResourceRequest(RMAppManager.java:385) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:345)  at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:271)
 at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitA pplication(ClientRMService.java:538)
            at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolP                                                                                                             BServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:188)
            at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:323)
            at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
            at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
            at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
            at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:415)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)

有人可以告诉我这里有什么问题吗？

Answer 1

该错误表明YARN环境中节点上的内存不足。可能是节点机器/ VM没有足够的内存，或者yarn.nodemanager.resource.memory-mb中的yarn-site.xml值设置为小于3072的值，但是您的流应用程序需要更多的内存。