java - mapper执行的判断

Question

定义映射器的帮助是否已执行，如果未执行，可能是出于什么原因。我将读取方式的输出从数据库写入到执行映射器的本地文件系统的文本文件。这里我给个代码

package org.myorg;

import java.io.*;
import java.util.*;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.logging.Level;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;


public class ParallelIndexation {

    public static class Map extends MapReduceBase implements
            Mapper<LongWritable, Text, Text, LongWritable> {
        private final static LongWritable zero = new LongWritable(0);
        private Text word = new Text();


        public void map(LongWritable key, Text value,
                OutputCollector<Text, LongWritable> output, Reporter reporter)
                throws IOException {

            Configuration conf = new Configuration();
            int CountComputers;
            FileInputStream fstream = new FileInputStream(
                    "/export/hadoop-1.0.1/bin/countcomputers.txt");
            BufferedReader br = new BufferedReader(new InputStreamReader(fstream));
            String result=br.readLine();
            CountComputers=Integer.parseInt(result);
            input.close();
            fstream.close();
            Connection con = null;
            Statement st = null;
                ResultSet rs = null;    
                String url = "jdbc:postgresql://192.168.1.8:5432/NexentaSearch";
                String user = "postgres";
                String password = "valter89";
            ArrayList<String> paths = new ArrayList<String>();
            try
            {
                con = DriverManager.getConnection(url, user, password);
                        st = con.createStatement();
                        rs = st.executeQuery("select path from tasks order by id");
                while (rs.next()) { paths.add(rs.getString(1)); };
                PrintWriter zzz = null;
                    try
                    {
                            zzz = new PrintWriter(new FileOutputStream("/export/hadoop-1.0.1/bin/readwaysfromdatabase.txt"));
                    }
                    catch(FileNotFoundException e)
                    {
                            System.out.println("Error");
                            System.exit(0);
                    }
                    for (int i=0; i<paths.size(); i++)
                {
                    zzz.println("paths[i]=" + paths.get(i) + "\n");
                    }
                    zzz.close();
            }
            catch (SQLException e) 
            {
                System.out.println("Connection Failed! Check output console");
                e.printStackTrace();
            }

但是，尽管在其中一个从属节点上，/export/hadoop-1.0.1/bin/readwaysfromdatabase.txt 文件并未创建。从这里是否可以看出，根本没有执行什么映射器？我还将输出带入程序的执行文件

args[0]=/export/hadoop-1.0.1/bin/input
13/04/22 14:00:53 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/04/22 14:00:53 INFO mapred.FileInputFormat: Total input paths to process : 0
13/04/22 14:00:54 INFO mapred.JobClient: Running job: job_201304221331_0003
13/04/22 14:00:55 INFO mapred.JobClient:  map 0% reduce 0%
13/04/22 14:01:12 INFO mapred.JobClient:  map 0% reduce 100%
13/04/22 14:01:17 INFO mapred.JobClient: Job complete: job_201304221331_0003
13/04/22 14:01:17 INFO mapred.JobClient: Counters: 15
13/04/22 14:01:17 INFO mapred.JobClient:   Job Counters 
13/04/22 14:01:17 INFO mapred.JobClient:     Launched reduce tasks=1
13/04/22 14:01:17 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=9079
13/04/22 14:01:17 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/04/22 14:01:17 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/04/22 14:01:17 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=7983
13/04/22 14:01:17 INFO mapred.JobClient:   File Output Format Counters 
13/04/22 14:01:17 INFO mapred.JobClient:     Bytes Written=0
13/04/22 14:01:17 INFO mapred.JobClient:   FileSystemCounters
13/04/22 14:01:17 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=21536
13/04/22 14:01:17 INFO mapred.JobClient:   Map-Reduce Framework
13/04/22 14:01:17 INFO mapred.JobClient:     Reduce input groups=0
13/04/22 14:01:17 INFO mapred.JobClient:     Combine output records=0
13/04/22 14:01:17 INFO mapred.JobClient:     Reduce shuffle bytes=0
13/04/22 14:01:17 INFO mapred.JobClient:     Reduce output records=0
13/04/22 14:01:17 INFO mapred.JobClient:     Spilled Records=0
13/04/22 14:01:17 INFO mapred.JobClient:     Total committed heap usage (bytes)=16252928
13/04/22 14:01:17 INFO mapred.JobClient:     Combine input records=0
13/04/22 14:01:17 INFO mapred.JobClient:     Reduce input records=0

我还在一个虚拟机上成功执行程序的文件中带了一个输出

12/10/28 10:41:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12/10/28 10:41:14 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/10/28 10:41:14 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/28 10:41:15 INFO mapred.JobClient: Running job: job_local_0001
12/10/28 10:41:15 INFO mapred.Task:  Using ResourceCalculatorPlugin : null
12/10/28 10:41:15 INFO mapred.MapTask: numReduceTasks: 1
12/10/28 10:41:15 INFO mapred.MapTask: io.sort.mb = 100
12/10/28 10:41:15 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/28 10:41:15 INFO mapred.MapTask: record buffer = 262144/327680
12/10/28 10:41:15 INFO mapred.MapTask: Starting flush of map output
12/10/28 10:41:16 INFO mapred.JobClient:  map 0% reduce 0%
12/10/28 10:41:17 INFO mapred.MapTask: Finished spill 0
12/10/28 10:41:17 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/10/28 10:41:18 INFO mapred.LocalJobRunner: file:/export/hadoop-1.0.1/bin/input/paths.txt:0+156
12/10/28 10:41:18 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
12/10/28 10:41:18 INFO mapred.Task:  Using ResourceCalculatorPlugin : null
12/10/28 10:41:18 INFO mapred.LocalJobRunner: 
12/10/28 10:41:18 INFO mapred.Merger: Merging 1 sorted segments
12/10/28 10:41:18 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 199 bytes
12/10/28 10:41:18 INFO mapred.LocalJobRunner: 
12/10/28 10:41:19 INFO mapred.JobClient:  map 100% reduce 0%
12/10/28 10:41:19 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
12/10/28 10:41:19 INFO mapred.LocalJobRunner: 
12/10/28 10:41:19 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
12/10/28 10:41:19 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to file:/export/hadoop-1.0.1/bin/output
12/10/28 10:41:21 INFO mapred.LocalJobRunner: reduce > reduce
12/10/28 10:41:21 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
12/10/28 10:41:22 INFO mapred.JobClient:  map 100% reduce 100%
12/10/28 10:41:22 INFO mapred.JobClient: Job complete: job_local_0001
12/10/28 10:41:22 INFO mapred.JobClient: Counters: 18
12/10/28 10:41:22 INFO mapred.JobClient:   File Input Format Counters 
12/10/28 10:41:22 INFO mapred.JobClient:     Bytes Read=156
12/10/28 10:41:22 INFO mapred.JobClient:   File Output Format Counters 
12/10/28 10:41:22 INFO mapred.JobClient:     Bytes Written=177
12/10/28 10:41:22 INFO mapred.JobClient:   FileSystemCounters
12/10/28 10:41:22 INFO mapred.JobClient:     FILE_BYTES_READ=9573
12/10/28 10:41:22 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=73931
12/10/28 10:41:22 INFO mapred.JobClient:   Map-Reduce Framework
12/10/28 10:41:22 INFO mapred.JobClient:     Reduce input groups=4
12/10/28 10:41:22 INFO mapred.JobClient:     Map output materialized bytes=203
12/10/28 10:41:22 INFO mapred.JobClient:     Combine output records=4
12/10/28 10:41:22 INFO mapred.JobClient:     Map input records=1
12/10/28 10:41:22 INFO mapred.JobClient:     Reduce shuffle bytes=0
12/10/28 10:41:22 INFO mapred.JobClient:     Reduce output records=4
12/10/28 10:41:22 INFO mapred.JobClient:     Spilled Records=8
12/10/28 10:41:22 INFO mapred.JobClient:     Map output bytes=189
12/10/28 10:41:22 INFO mapred.JobClient:     Total committed heap usage (bytes)=321527808
12/10/28 10:41:22 INFO mapred.JobClient:     Map input bytes=156
12/10/28 10:41:22 INFO mapred.JobClient:     Combine input records=0
12/10/28 10:41:22 INFO mapred.JobClient:     Map output records=4
12/10/28 10:41:22 INFO mapred.JobClient:     SPLIT_RAW_BYTES=98
12/10/28 10:41:22 INFO mapred.JobClient:     Reduce input records=0

@ChrisWhite 我在命令的帮助下运行了程序

./hadoop jar /export/hadoop-1.0.1/bin/ParallelIndexation.jar org.myorg.ParallelIndexation /export/hadoop-1.0.1/bin/input /export/hadoop-1.0.1/bin/output -D mapred.map.tasks=1 1> resultofexecute.txt 2&>1 

我在一个集群中有 4 个节点，其中一个主节点，一个用于 secondarynamenode 和 2 个从属节点。

Answer 1

为您的作业安排了多少个 map task ，您的集群有多大？如果说您的作业仅运行 4 个映射任务和一个具有 32 个节点的集群，那么 28/32 个节点可能不会有任何输出(因为没有映射任务在这些节点上运行)。

您可以通过 Job Tracker Web UI 查看有关构成您的作业的 map task 数量以及这些作业计划在何处运行的信息。

奇怪的是，您的第一次运行转储没有显示任何已启动的 map 作业，只是减少了任务:

13/04/22 14:01:17 INFO mapred.JobClient:     Launched reduce tasks=1

而且 map 输入/输出记录也没有计数器，因此您运行此作业的方式有些奇怪 - 您能否分享您用于启 Action 业的完整命令行以及可能的驱动程序代码配置并运行作业？