hadoop - hadoop字数示例

Question

我是hadoop的新手，我刚刚安装了hadoop 2.6。

看来系统启动正常。我正在尝试运行单词计数示例，但问题是一切似乎都在运行，输出文件夹是用2个文件创建的:

-rw-r--r-- 1 yoni supergroup 0 2016-04-30 02:11 / user / yoni / output100 / _SUCCESS
-rw-r--r-- 1 yoni supergroup 0 2016-04-30 02:11 / user / yoni / output100 / part-r-00000

但文件为空，部分为r-00000。问题是我不知道要寻找问题，

这是作业的日志:

16/04/30 20:30:33 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/04/30 20:30:34 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
16/04/30 20:30:34 INFO input.FileInputFormat: Total input paths to process : 1
16/04/30 20:30:34 INFO mapreduce.JobSubmitter: number of splits:1
16/04/30 20:30:34 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1461971181442_0005
16/04/30 20:30:34 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
16/04/30 20:30:34 INFO impl.YarnClientImpl: Submitted application application_1461971181442_0005
16/04/30 20:30:34 INFO mapreduce.Job: The url to track the job: http://yoni-Lenovo-Z40-70:8088/proxy/application_1461971181442_0005/
16/04/30 20:30:34 INFO mapreduce.Job: Running job: job_1461971181442_0005
16/04/30 20:30:41 INFO mapreduce.Job: Job job_1461971181442_0005 running in uber mode : false
16/04/30 20:30:41 INFO mapreduce.Job:  map 0% reduce 0%
16/04/30 20:30:46 INFO mapreduce.Job:  map 100% reduce 0%
16/04/30 20:30:51 INFO mapreduce.Job:  map 100% reduce 100%
16/04/30 20:30:52 INFO mapreduce.Job: Job job_1461971181442_0005 completed successfully
16/04/30 20:30:52 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=6
        FILE: Number of bytes written=211511
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=170
        HDFS: Number of bytes written=86
        HDFS: Number of read operations=6
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Launched reduce tasks=1
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=2923
        Total time spent by all reduces in occupied slots (ms)=2526
        Total time spent by all map tasks (ms)=2923
        Total time spent by all reduce tasks (ms)=2526
        Total vcore-seconds taken by all map tasks=2923
        Total vcore-seconds taken by all reduce tasks=2526
        Total megabyte-seconds taken by all map tasks=2993152
        Total megabyte-seconds taken by all reduce tasks=2586624
    Map-Reduce Framework
        Map input records=1
        Map output records=0
        Map output bytes=0
        Map output materialized bytes=6
        Input split bytes=116
        Combine input records=0
        Combine output records=0
        Reduce input groups=0
        Reduce shuffle bytes=6
        Reduce input records=0
        Reduce output records=0
        Spilled Records=0
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=166
        CPU time spent (ms)=1620
        Physical memory (bytes) snapshot=426713088
        Virtual memory (bytes) snapshot=3818450944
        Total committed heap usage (bytes)=324009984
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=54
    File Output Format Counters 
        Bytes Written=86
16/04/30 20:30:52 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/04/30 20:30:52 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
16/04/30 20:30:52 INFO input.FileInputFormat: Total input paths to process : 1
16/04/30 20:30:52 INFO mapreduce.JobSubmitter: number of splits:1
16/04/30 20:30:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1461971181442_0006
16/04/30 20:30:52 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
16/04/30 20:30:52 INFO impl.YarnClientImpl: Submitted application application_1461971181442_0006
16/04/30 20:30:52 INFO mapreduce.Job: The url to track the job: http://yoni-Lenovo-Z40-70:8088/proxy/application_1461971181442_0006/
16/04/30 20:30:52 INFO mapreduce.Job: Running job: job_1461971181442_0006
16/04/30 20:31:01 INFO mapreduce.Job: Job job_1461971181442_0006 running in uber mode : false
16/04/30 20:31:01 INFO mapreduce.Job:  map 0% reduce 0%
16/04/30 20:31:07 INFO mapreduce.Job:  map 100% reduce 0%
16/04/30 20:31:12 INFO mapreduce.Job:  map 100% reduce 100%
16/04/30 20:31:13 INFO mapreduce.Job: Job job_1461971181442_0006 completed successfully
16/04/30 20:31:13 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=6
        FILE: Number of bytes written=210495
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=216
        HDFS: Number of bytes written=0
        HDFS: Number of read operations=7
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Launched reduce tasks=1
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=3739
        Total time spent by all reduces in occupied slots (ms)=3133
        Total time spent by all map tasks (ms)=3739
        Total time spent by all reduce tasks (ms)=3133
        Total vcore-seconds taken by all map tasks=3739
        Total vcore-seconds taken by all reduce tasks=3133
        Total megabyte-seconds taken by all map tasks=3828736
        Total megabyte-seconds taken by all reduce tasks=3208192
    Map-Reduce Framework
        Map input records=0
        Map output records=0
        Map output bytes=0
        Map output materialized bytes=6
        Input split bytes=130
        Combine input records=0
        Combine output records=0
        Reduce input groups=0
        Reduce shuffle bytes=6
        Reduce input records=0
        Reduce output records=0
        Spilled Records=0
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=125
        CPU time spent (ms)=1010
        Physical memory (bytes) snapshot=427823104
        Virtual memory (bytes) snapshot=3819626496
        Total committed heap usage (bytes)=324534272
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=86
    File Output Format Counters 
        Bytes Written=0

我正在运行hadoop安装随附的wordcount示例

hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep / user / yoni / input / user / yoni / output101'dfs [a-z。] +'

以及所有基本教程中的伪分布式模式下的设置

Answer 1

在此示例中，应将xml下的所有hadoop-2.6.4/etc/hadoop文件放入正确的用户主目录中的名为“input”的HDFS文件夹中，此处为yoni。

因此，首先通过探索HDFS(默认情况)来检查http://localhost:50070守护进程的状态。

其次，通过bin/hdfs dfs -ls /user/yoni/input或bin/hdfs fsck / -files -blocks检查文件的状态。

如果一切顺利，它应该可以工作。

Hadoop MapReduce Next Generation - Setting up a Single Node Cluster