hadoop - MapReduce驱动的addInputPath错误

Question

我在 MapReduce 驱动程序的 addInputPath 方法中遇到错误。错误是

"The method addInputPath(Job, Path) in the type FileInputFormat is not applicable for the arguments (JobConf, Path)"

这是我的驱动程序代码:

package org.myorg;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class WordCount extends Configured implements Tool{
    public int run(String[] args) throws Exception
    {
          //creating a JobConf object and assigning a job name for identification purposes
          JobConf conf = new JobConf(getConf(), org.myorg.WordCount.class);
          conf.setJobName("WordCount");

          //Setting configuration object with the Data Type of output Key and Value
          conf.setOutputKeyClass(Text.class);
          conf.setOutputValueClass(IntWritable.class);

          //Providing the mapper and reducer class names
          conf.setMapperClass(WordCountMapper.class);
          conf.setReducerClass(WordCountReducer.class);

          //the hdfs input and output directory to be fetched from the command line
          **FileInputFormat.addInputPath(conf, new Path(args[0]));**
          FileOutputFormat.setOutputPath(conf, new Path(args[1]));

          JobClient.runJob(conf);
          return 0;
    }

    public static void main(String[] args) throws Exception
    {
          int res = ToolRunner.run(new Configuration(), new WordCount(),args);
          System.exit(res);
    }
}

我导入了正确的 org.apache.hadoop.mapred.FileOutputFormat。

我的 WordCountMapper 正确实现了 Mapper。

FileOutputFormat.setOutputPath 工作正常。

为什么 addInputhPaths 会抛出错误？

Answer 1

问题是您混合了旧 API (.mapred.) 和新 API (.mapreduce.)。这两个 API 不兼容。

我建议您使用新 API 中的所有对象，而不使用旧 API 中的任何对象。也就是说，不要使用 JobConf 或 JobClient。请改用 Job 和 Configuration。并确保您从包含 .mapreduce. 的导入中使用 Mapper、Reducer 等。不是 .mapred..