hadoop - 如何查看我的mapreduce作业是否在多个节点上执行？

Question

我已经在集群中运行了Map Reduce作业。我在HDInsight上创建了一个作业，其中包含2个namenode和4个datanode。

我没有设置 map task 和 reduce task 的数量。执行mapreduce作业后，得到的结果如下。

我注意到启动的reduce任务的数量是1.dose，这意味着我的工作仅在一个节点上执行吗？如何查看此作业中使用了多少个节点？

File System Counters
                FILE: Number of bytes read=2209390166
                FILE: Number of bytes written=3314494070
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                WASB: Number of bytes read=1084887535
                WASB: Number of bytes written=1205106549
                WASB: Number of read operations=0
                WASB: Number of large read operations=0
                WASB: Number of write operations=0
        Job Counters
                Launched map tasks=2
                Launched reduce tasks=1
                Rack-local map tasks=2
                Total time spent by all maps in occupied slots (ms)=148221
                Total time spent by all reduces in occupied slots (ms)=302038
                Total time spent by all map tasks (ms)=148221
                Total time spent by all reduce tasks (ms)=151019
                Total vcore-seconds taken by all map tasks=148221
                Total vcore-seconds taken by all reduce tasks=151019
                Total megabyte-seconds taken by all map tasks=113833728
                Total megabyte-seconds taken by all reduce tasks=231965184
        Map-Reduce Framework
                Map input records=3820642
                Map output records=3820642
                Map output bytes=1092815223
                Map output materialized bytes=1104695065
                Input split bytes=286
                Combine input records=0
                Combine output records=0
                Reduce input groups=200
                Reduce shuffle bytes=1104695065
                Reduce input records=3820642
                Reduce output records=3820642
                Spilled Records=11461926
                Shuffled Maps =2
                Failed Shuffles=0
                Merged Map outputs=2
                GC time elapsed (ms)=16229
                CPU time spent (ms)=225140
                Physical memory (bytes) snapshot=1377296384
                Virtual memory (bytes) snapshot=5068787712
                Total committed heap usage (bytes)=1175453696
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=1084887106
        File Output Format Counters
                Bytes Written=68771556

Answer 1

是的，您将reduce的数量设置为1，输出reduce为1，可以看到它Map输入记录= 3820642
map 输出记录= 3820642
映射输出字节= 1092815223
映射输出实例化字节= 1104695065
输入分割字节= 286
合并输入记录= 0
合并输出记录= 0
减少输入组= 200
减少随机播放字节= 1104695065
减少输入记录= 3820642
减少输出记录= 3820642