从属节点上未发生 Hadoop Mapreduce 任务

Question

我正在按照 this tutorial 在双节点 Linux 集群(Ubuntu 虚拟机)中尝试 Hadoop map-reduce .

当我运行 wordcount map reduce 程序时，该任务并未在从站上运行。你能帮忙找出问题所在吗？

请找到我的日志和输出文件。

ma​​ster 的 Jps 输出:

hduser@master:/usr/local/hadoop$ jps
8056 NodeManager
8696 Jps
7471 NameNode
7592 DataNode
7793 SecondaryNameNode
7933 ResourceManager

slave的Jps输出:

hduser@slave:/usr/local/hadoop$ jps
3634 NodeManager
3518 DataNode
3722 Jps

运行 jar 文件的输出:

hduser@master:/usr/local/hadoop$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /user/hduser/WordCount/ /user/hduser/WordCount/MultiNode_Output
15/11/30 04:07:16 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
15/11/30 04:07:16 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
15/11/30 04:07:17 INFO input.FileInputFormat: Total input paths to process : 3
15/11/30 04:07:17 INFO mapreduce.JobSubmitter: number of splits:3
15/11/30 04:07:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1403989830_0001
15/11/30 04:07:18 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
15/11/30 04:07:18 INFO mapreduce.Job: Running job: job_local1403989830_0001
15/11/30 04:07:18 INFO mapred.LocalJobRunner: OutputCommitter set in config null
15/11/30 04:07:18 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
15/11/30 04:07:18 INFO mapred.LocalJobRunner: Waiting for map tasks
15/11/30 04:07:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1403989830_0001_m_000000_0
15/11/30 04:07:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
15/11/30 04:07:18 INFO mapred.MapTask: Processing split: hdfs://master:54310/user/hduser/WordCount/pg4300.txt:0+1573151
15/11/30 04:07:19 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
15/11/30 04:07:19 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
15/11/30 04:07:19 INFO mapred.MapTask: soft limit at 83886080
15/11/30 04:07:19 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
15/11/30 04:07:19 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
15/11/30 04:07:19 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
15/11/30 04:07:19 INFO mapreduce.Job: Job job_local1403989830_0001 running in uber mode : false
15/11/30 04:07:19 INFO mapreduce.Job:  map 0% reduce 0%
15/11/30 04:07:19 INFO input.LineRecordReader: Found UTF-8 BOM and skipped it
15/11/30 04:07:20 INFO mapred.LocalJobRunner: 
15/11/30 04:07:20 INFO mapred.MapTask: Starting flush of map output
15/11/30 04:07:20 INFO mapred.MapTask: Spilling map output
15/11/30 04:07:20 INFO mapred.MapTask: bufstart = 0; bufend = 2601881; bufvoid = 104857600
15/11/30 04:07:20 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25142496(100569984); length = 1071901/6553600
15/11/30 04:07:22 INFO mapred.MapTask: Finished spill 0
15/11/30 04:07:22 INFO mapred.Task: Task:attempt_local1403989830_0001_m_000000_0 is done. And is in the process of committing
15/11/30 04:07:22 INFO mapred.LocalJobRunner: map
15/11/30 04:07:22 INFO mapred.Task: Task 'attempt_local1403989830_0001_m_000000_0' done.
15/11/30 04:07:22 INFO mapred.LocalJobRunner: Finishing task: attempt_local1403989830_0001_m_000000_0
15/11/30 04:07:22 INFO mapred.LocalJobRunner: Starting task: attempt_local1403989830_0001_m_000001_0
15/11/30 04:07:22 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
15/11/30 04:07:22 INFO mapred.MapTask: Processing split: hdfs://master:54310/user/hduser/WordCount/5000-8.txt:0+1428841
…
…
15/11/30 04:07:27 INFO mapred.LocalJobRunner: reduce task executor complete.
15/11/30 04:07:27 INFO mapreduce.Job:  map 100% reduce 100%
15/11/30 04:07:28 INFO mapreduce.Job: Job job_local1403989830_0001 completed successfully
15/11/30 04:07:28 INFO mapreduce.Job: Counters: 38
    File System Counters
        FILE: Number of bytes read=4010919
        FILE: Number of bytes written=8405245
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=11928267
        HDFS: Number of bytes written=883509
        HDFS: Number of read operations=37
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=6
    Map-Reduce Framework
        Map input records=78578
        Map output records=629920
        Map output bytes=6083556
        Map output materialized bytes=1462980
        Input split bytes=352
        Combine input records=629920
        Combine output records=101397
        Reduce input groups=82616
        Reduce shuffle bytes=1462980
        Reduce input records=101397
        Reduce output records=82616
        Spilled Records=202794
        Shuffled Maps =3
        Failed Shuffles=0
        Merged Map outputs=3
        GC time elapsed (ms)=433
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
        Total committed heap usage (bytes)=657997824
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=3676562
    File Output Format Counters 
        Bytes Written=883509

hadoop-hduser-datanode-slave.log:

   2243 2015-11-30 04:13:56,747 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: slave/192.168.56.102:54310. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
   2244 2015-11-30 04:13:56,748 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: slave/192.168.56.102:54310

yarn-hduser-nodemanager-slave.log:

991 2015-11-30 04:09:39,366 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 127.0.0.1/127.0.0.1:8031. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
    992 2015-11-30 04:09:40,367 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 127.0.0.1/127.0.0.1:8031. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
    993 2015-11-30 04:09:41,368 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 127.0.0.1/127.0.0.1:8031. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
    994 2015-11-30 04:09:42,369 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 127.0.0.1/127.0.0.1:8031. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
    995 2015-11-30 04:09:43,370 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 127.0.0.1/127.0.0.1:8031. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

当我转到 http://localhost:50070/ 时，我只能看到一个 Live Node，它是我的 master。

Answer 1

我想问题应该解决了，如果所有下面的测试都ok:

• 停止集群(执行 jps 以检查 hadoop 是否未运行)
• 更改 /etc/hosts 文件，其中 localIpOfMaster 应该类似于 127.0.0.1:

<localIpOfMaster> localhost
<publicIpOfMaster> master
<ipOfSlave> slave1

• 在 hadoop 配置的 slaves 文件中，对从站使用相同的名称(这里是 slave1)。
• 确保您可以 无需密码 ssh到从站，使用与 /etc/hosts 文件和 slaves 文件中相同的名称(这里是 ssh slave1)
• 再次启动集群并检查 jps 是否一切正常。再次检查日志以验证没有连接错误

希望对您有所帮助。如果没有，因为您发布该问题已经有一段时间了，问题可能已经解决，请与我们分享您可能找到的解决方案，以便面临相同问题的其他人可以找到它。