hadoop - yarn : what subsystem connecting to port 44874

Question

我正在尝试在 YARN 上运行我的 MR 作业。节点 3 上的其中一个用户日志中存在此错误:

2014-10-10 00:57:16,965 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2014-10-10 00:57:16,965 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1412895371072_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@69d5af30)
2014-10-10 00:57:17,330 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2014-10-10 00:57:18,547 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: node03/127.0.1.1:44874. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2014-10-10 00:57:19,548 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: node03/127.0.1.1:44874. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
...
2014-10-10 00:57:27,558 INFO [main] org.apache.hadoop.ipc.Client: Retrying connect to server: node03/127.0.1.1:44874. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2014-10-10 00:57:27,562 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.net.ConnectException: Call From node03/127.0.1.1 to node03:44874 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
        at org.apache.hadoop.ipc.Client.call(Client.java:1415)
        at org.apache.hadoop.ipc.Client.call(Client.java:1364)
        at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:231)
        at com.sun.proxy.$Proxy9.getTask(Unknown Source)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:137)
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
        at org.apache.hadoop.ipc.Client.call(Client.java:1382)
        ... 4 more

2014-10-10 00:57:27,564 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
2014-10-10 00:57:27,566 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
:

我在所有节点上都有相同的配置。我找不到指定端口 44874 的任何地方。这个错误实际上说明了什么？

Answer 1

如果“半随机”是指完全随机，而“无法真正配置”是指在强化环境中使用时未记录且完全错误 - 你是对的。

问题在于 map-reduce 作业正在使用动态端口。 Horton 当然没有记录为什么要创建随机端口。

到目前为止的答案是:禁用防火墙或允许每个数据节点上的高范围 (32768-65535)。我还在寻找为什么会出现这种情况。