java - 如何将 hadoop 输入格式设置为 NLineInputFormat？

Question

我试图限制每个映射器获得的行数。我的代码是这样的:

    package com.iathao.mapreduce;

    import java.io.IOException;
    import java.net.MalformedURLException;

    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapred.lib.NLineInputFormat;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    import org.apache.regexp.RESyntaxException;

    import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;

    public class Main {


    public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException, RESyntaxException {

    try {
        if (args.length != 2) {
            System.err.println("Usage: NewMaxTemperature <input path> <output path>");
            System.exit(-1);
        }
        Job job = new Job();
        job.setJarByClass(Main.class);
        job.getConfiguration().set("mapred.max.map.failures.percent", "100");
        // job.getConfiguration().set("mapred.map.max.attempts", "10");
        //NLineInputFormat. .setNumLinesPerSplit(job, 1);
        job.setInputFormatClass(NLineInputFormat.class);

在样本的最后一行 (job.setInputFormatClass(NLineInputFormat.class);) 我得到以下错误:

The method setInputFormatClass(Class<? extends InputFormat>) in the type Job is not applicable for the arguments (Class<NLineInputFormat>)

我是不是弄错了 NLineInputFormat 类？

Answer 1

您正在混合使用新旧 API。

import org.apache.hadoop.mapred.lib.NLineInputFormat;
import org.apache.hadoop.mapreduce.Job;

根据《Hadoop:权威指南》

The new API is in the org.apache.hadoop.mapreduce package (and subpackages). The old API can still be found in org.apache.hadoop.mapred.

如果您计划使用新的 API，请使用 org.apache.hadoop.mapreduce 包中的 NLineInputFormat。