apache - 我在测试hadoop多节点集群时遇到了一些问题

Question

你好我正在使用带有 2 个节点的 hadoop 0.23.9当我尝试单节点时它工作正常但是当我尝试测试字数统计作业时，我得到以下结果，但我不知道错误在哪里

我在两台 PC 上都使用 ubuntu 12.04 LTS

        mapred-site.xml


        <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
        </property>

        </configuration>

         yarn-site.xml


<configuration>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce.shuffle</value>
  </property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>

<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>


 <property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
  </property>

<property>
 <name>mapreduce.job.tracker</name>
  <value>master</value>
 </property>
<configuration>

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce.shuffle</value>
  </property>




<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>

<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>

<property>
<name>yarn.nodemanager.address</name>
<value>0.0.0.0:8042</value>
</property>

</configuration>

这里是ubuntu的hosts文件

    127.0.0.1       localhost
    127.0.1.1       fatima1-HP-ProBook-4530s

192.168.1.10    master
192.168.1.11    slave

#the following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters









        14/04/25 15:41:39 INFO input.FileInputFormat: Total input paths to process : 7
        14/04/25 15:41:39 WARN snappy.LoadSnappy: Snappy native library not loaded
        14/04/25 15:41:39 INFO mapreduce.JobSubmitter: number of splits:7
        14/04/25 15:41:39 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
        14/04/25 15:41:39 WARN conf.Configuration: mapred.output.value.class is deprecated.    Instead, use mapreduce.job.output.value.class
        14/04/25 15:41:39 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
        14/04/25 15:41:39 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
        14/04/25 15:41:39 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
        14/04/25 15:41:39 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
        14/04/25 15:41:39 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
        14/04/25 15:41:39 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
        14/04/25 15:41:39 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
        14/04/25 15:41:39 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
        14/04/25 15:41:39 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
        14/04/25 15:41:39 INFO mapred.ResourceMgrDelegate: Submitted application application_1398429615167_0001 to ResourceManager at master/192.168.1.10:8032
        14/04/25 15:41:39 INFO mapreduce.Job: The url to track the job: `http://master:8088/proxy/application_1398429615167_0001/`
        14/04/25 15:41:39 INFO mapreduce.Job: Running job: job_1398429615167_0001
        14/04/25 15:41:46 INFO mapreduce.Job: Job job_1398429615167_0001 running in uber mode : false
        14/04/25 15:41:46 INFO mapreduce.Job:  map 0% reduce 0%
        14/04/25 15:41:47 INFO mapreduce.Job: Task Id : attempt_1398429615167_0001_m_000006_0, Status : FAILED
    Container launch failed for container_1398429615167_0001_01_000008 : java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:157)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
    Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
        at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
        at com.sun.proxy.$Proxy28.startContainer(Unknown Source)
        at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104)
    ... 5 more
Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:238)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1194)
    at org.apache.hadoop.ipc.Client.call(Client.java:1074)
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
    ... 7 more
Caused by: java.net.UnknownHostException
    ... 11 more

14/04/25 15:41:49 INFO mapreduce.Job: Task Id : attempt_1398429615167_0001_m_000006_1, Status : FAILED
Container launch failed for container_1398429615167_0001_01_000009 : java.lang.reflect.UndeclaredThrowableException
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:157)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
    at com.sun.proxy.$Proxy28.startContainer(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104)
    ... 5 more
Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  http://wiki.apache.org/hadoop/UnknownHost
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:238)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1194)
    at org.apache.hadoop.ipc.Client.call(Client.java:1074)
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
    ... 7 more
Caused by: java.net.UnknownHostException
    ... 11 more

14/04/25 15:41:51 INFO mapreduce.Job: Task Id : attempt_1398429615167_0001_m_000006_2, Status : FAILED
Container launch failed for container_1398429615167_0001_01_000010 : java.lang.reflect.UndeclaredThrowableException
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:157)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
    at com.sun.proxy.$Proxy28.startContainer(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104)
    ... 5 more
Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""fatima-HP-ProBook-4520s":45527; java.net.UnknownHostException; For more details see:  `http://wiki.apache.org/hadoop/UnknownHost`
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:238)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1194)
    at org.apache.hadoop.ipc.Client.call(Client.java:1074)
    at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
    ... 7 more
Caused by: java.net.UnknownHostException
    ... 11 more

14/04/25 15:41:53 INFO mapreduce.Job:  map 14% reduce 0%
14/04/25 15:41:53 INFO mapreduce.Job: Job job_1398429615167_0001 failed with state FAILED due to: Task failed task_1398429615167_0001_m_000006
Job failed as tasks failed. failedMaps:1 failedReduces:0

14/04/25 15:41:53 INFO mapreduce.Job: Counters: 5
    Job Counters 
        Launched map tasks=6
        Other local map tasks=3
        Data-local map tasks=7
        Total time spent by all maps in occupied slots (ms)=0
        Total time spent by all reduces in occupied slots (ms)=0

Answer 1

您的主机名在/etc/hosts 中设置不正确。您应该完全删除以下行。

127.0.1.1       fatima1-HP-ProBook-4530s

然后在本地主机条目之后添加您的实际 IP 地址。您的/etc/hosts 文件应该如下所示。注意:我假设这是主人，我可能是错的，在这种情况下用实际的 IP 地址替换它。还要检查目标机器的/etc/hosts 文件。

    127.0.0.1       localhost
    192.168.1.10    fatima1-HP-ProBook-4530s

192.168.1.10    master
192.168.1.11    slave

the following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

完成后，您需要重启机器并重新启动 hadoop

编辑:您的 core-site.xml 必须具有正确的主/从条目，如下所示。您可能在 core-site.xml 中设置了其他属性，这很好。但在分布式模式下，您需要指定一个 fs.default.name 作为主控或从属。

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://master:9000</value>
  </property>
</configuration>