java - hadoop map reduce作业没有输出

Question

我在 Netbeans 中编写 MapReduce 作业并生成(也在 NB 中)一个 jar 文件。当我尝试在 hadoop(版本 1.2.1)中执行此作业时，我执行此命令:

$ hadoop jar job.jar org.job.mainClass /home/user/in.txt /home/user/outdir

此命令不显示任何错误，但不创建 outdir、outfiles、...

这是我的工作代码:

映射器

public class Mapper extends MapReduceBase implements org.apache.hadoop.mapred.Mapper<LongWritable, Text, Text, IntWritable> {

            private final IntWritable one = new IntWritable(1);
            private Text company = new Text("");


            @Override
            public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
                company.set(value.toString());
                output.collect(value, one);

            }

        }

reducer

public class Reducer extends MapReduceBase implements org.apache.hadoop.mapred.Reducer<Text, IntWritable, Text, IntWritable> {

    @Override
    public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {

        int sum = 0;
        while (values.hasNext()){
            sum++;
            values.next();
        }

        output.collect(key, new IntWritable(sum));
    }
}

主要

 public static void main(String[] args) {

    JobConf configuration = new JobConf(CdrMR.class);
    configuration.setJobName("Dedupe companies");
    configuration.setOutputKeyClass(Text.class);
    configuration.setOutputValueClass(IntWritable.class);
    configuration.setMapperClass(Mapper.class);
    configuration.setReducerClass(Reducer.class);
    configuration.setInputFormat(TextInputFormat.class);
    configuration.setOutputFormat(TextOutputFormat.class);
    FileInputFormat.setInputPaths(configuration, new Path(args[0]));
    FileOutputFormat.setOutputPath(configuration, new Path(args[1]));

}

输入文件格式如下:

name1
name2
name3
...

还说我在没有 root 权限的虚拟机 (Ubuntu 12.04) 中执行 hadoop。 Hadoop 是否在不同的目录中执行作业和存储输出文件？

Answer 1

根据 this article您需要使用此方法提交您的 JobConf:

JobClient.runJob(configuration);