gpt4 book ai didi

java - Hadoop 作业在减少时失败,Java.io.IOException : Type mismatch in value from map

转载 作者:行者123 更新时间:2023-12-01 15:14:53 24 4
gpt4 key购买 nike

我的Map类是

 public static class MapClass extends Mapper<LongWritable, Text, Text, LongWritable> {

public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
// your map code goes here
String[] fields = value.toString().split(",");
String year = fields[1];
String claims = fields[8];

if (claims.length() > 0 && (!claims.startsWith("\""))) {
context.write(new Text(year), new LongWritable(Long.parseLong(claims)));
}
}
}

我的Reduce类看起来像

   public static class Reduce extends Reducer<Text, LongWritable, Text, Text> {


public void reduce(Text key, Iterable<LongWritable> values, Context context) throws IOException, InterruptedException {
// your reduce function goes here
context.write(key, new Text("hello"));
}
}

数据集看起来像

3070801,1963,1096,,"BE","",,1,,269,6,69,,1,,0,,,,,,,
3070802,1963,1096,,"US","TX",,1,,2,6,63,,0,,,,,,,,,

当我使用配置运行程序时

        Job job = new Job();
job.setJarByClass(TopKRecords.class);

job.setMapperClass(MapClass.class);
job.setReducerClass(Reduce.class);

FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

job.setJobName("TopKRecords");
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);

我看到错误为

java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1019)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at com.hadoop.programs.TopKRecords$MapClass.map(TopKRecords.java:35)
at com.hadoop.programs.TopKRecords$MapClass.map(TopKRecords.java:26)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

这里出了什么问题?

我在这里没有看到任何不匹配的原因

Mapper<LongWritable, Text, Text, LongWritable>
Reducer<Text, LongWritable, Text, Text>

更新

设置以下内容后,一切开始工作

        job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(LongWritable.class);

最佳答案

在设置过程中您还需要以下行:

job.setMapOutputValueClass(LongWritable.class);

来自 Hadoop 20.2 Javadoc :

This allows the user to specify the map output value class to be different than the final output value class.

为了清楚起见,您还可以添加:

job.setMapOutputKeyClass(Text.class);

但在这种情况下没有必要。

关于java - Hadoop 作业在减少时失败,Java.io.IOException : Type mismatch in value from map,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/11761135/

24 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com