hadoop - 无法在hadoop的WordCount作业中配置 reducer 的数量

Question

我在Linum Machine中使用单节点群集-Hadoop-2.7.0。
我的WordCount Job代码在1个reducer上运行良好。
但是，如果我增加 reducer ，则无法正常工作。
它显示以下错误:

15/05/25 21:15:10 INFO util.NativeCodeLoader: Loaded the native-hadoop library
15/05/25 21:15:10 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
15/05/25 21:15:10 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
15/05/25 21:15:10 WARN snappy.LoadSnappy: Snappy native library is available
15/05/25 21:15:10 INFO snappy.LoadSnappy: Snappy native library loaded
15/05/25 21:15:10 INFO mapred.FileInputFormat: Total input paths to process : 1
15/05/25 21:15:10 INFO mapred.JobClient: Running job: job_local_0001
15/05/25 21:15:11 INFO util.ProcessTree: setsid exited with exit code 0
15/05/25 21:15:11 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5f1fd699
15/05/25 21:15:11 INFO mapred.MapTask: numReduceTasks: 1
15/05/25 21:15:11 INFO mapred.MapTask: io.sort.mb = 100
15/05/25 21:15:11 INFO mapred.MapTask: data buffer = 79691776/99614720
15/05/25 21:15:11 INFO mapred.MapTask: record buffer = 262144/327680

15/05/25 21:15:11 WARN mapred.LocalJobRunner: job_local_0001
java.io.IOException: Illegal partition for am (1)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1073)
    at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:592)
    at WordMapper.map(WordMapper.java:24)
    at WordMapper.map(WordMapper.java:1)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)

我的getPartition方法如下所示:

public int getPartition(Text key, IntWritable value, int numRedTasks) {
        String s = key.toString();
        if(s.length() == 1)
        {
            return 0;
        }
        else if(s.length() == 2)
        {
            return 1;
        }
        else if(s.length() == 3)
        {
            return 2;
        }
        else
            return 3;
    }

在WordCount.class文件中运行方法:

if(input.length < 2)
    {
        System.out.println("Please provide valid input");
        return -1;
    }
    else
    {
        JobConf config = new JobConf();
        FileInputFormat.setInputPaths(config, new Path(input[0]));
        FileOutputFormat.setOutputPath(config, new Path(input[1]));
        config.setMapperClass(WordMapper.class);
        config.setReducerClass(WordReducer.class);
        config.setNumReduceTasks(4);
        config.setPartitionerClass(MyPartitioner.class);
        config.setMapOutputKeyClass(Text.class);
        config.setMapOutputValueClass(IntWritable.class);
        config.setOutputKeyClass(Text.class);
        config.setOutputValueClass(IntWritable.class);
        JobClient.runJob(config);
    }
return 0;

}

我的Mapper和Reducer代码很好，因为带有1个reducer的Wordcount Job运行正常。
有人能弄清楚吗？

Answer 1

这可能是由于可以在其中设置高default_parallel导致Pig操作失败。

谢谢，
谢勒什。