java.lang.Exception : java. io.IOException:设置 hadoop 和 hbase 时值类错误

Question

我是 Hadoop 和 Hbase 的新手，我正在尝试让它们协同工作。我构建了一个 .java 并获得了一个 .jar 存档，没有任何错误。无论如何，在启动程序时出现此错误:

    java.lang.Exception: java.io.IOException: wrong value class: class org.apache.hadoop.hbase.client.Put is not class org.apache.hadoop.io.IntWritable
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
    Caused by: java.io.IOException: wrong value class: class org.apache.hadoop.hbase.client.Put is not class org.apache.hadoop.io.IntWritable
at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:194)
at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1378)
at org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write(Task.java:1695)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
at TweetSentiment$ClassificationCounterReducer.reduce(TweetSentiment.java:131)
at TweetSentiment$ClassificationCounterReducer.reduce(TweetSentiment.java:114)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1716)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:844)

我不明白为什么。

我已经尝试了一些不同版本的 Hadoop 和 Hbase，但似乎不起作用。这是我的代码:

public static class ClassificationCounterReducer extends TableReducer<IntWritable,IntWritable,IntWritable> {

        //private IntWritable result = new IntWritable();

        public void reduce(IntWritable classification, Iterable<IntWritable> values, Context context)
                throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable val : values) {
                sum += val.get();
            }

            Put put = new Put(Bytes.toBytes(classification.toString()));

            put.addColumn( Bytes.toBytes("number"), Bytes.toBytes(""), Bytes.toBytes(sum) );
            context.write(classification, put);
        }
    }

    private static final String OUTPUT_TABLE = "sentiment";

    public int run(String[] args) throws Exception  {

        Job job = Job.getInstance(getConf(), "Sentiment Count");

        job.setJarByClass(TweetSentiment.class);
        job.setMapperClass(ClassificatorMapper.class);
        job.setCombinerClass(ClassificationCounterReducer.class);
        job.setReducerClass(ClassificationCounterReducer.class);

        TableMapReduceUtil.initTableReducerJob(
                OUTPUT_TABLE,
                TweetSentiment.ClassificationCounterReducer.class,
                job);

        job.setMapOutputKeyClass(IntWritable.class);
        job.setMapOutputValueClass(IntWritable.class);

        FileInputFormat.addInputPath(job, new Path(args[0]));

        System.exit(job.waitForCompletion(true) ? 0 : 1);

        return 0;
    }

    public static void main(String[] args) throws Exception {
        int res = ToolRunner.run(new HBaseConfiguration(), new TweetSentiment(), args);
        System.exit(res);
    }

该代码预计将对 Twitter 的推文进行情绪分析。映射器应该从 HDFS 读取推文，对它们进行分类并将结果发送到组合器/缩减器。 reducer 应该首先计数，然后将它们存储在一个 hbase 表中，该表有两个条目，分别称为“Sentiment”和“Number”。

Answer 1

您的类型不匹配 - 您正在输出 context.write(IntWritable, Put) ，但是你的 reducer 类扩展了 TableReducer<IntWritable,IntWritable,IntWritable> .

将第一行改为:

public static class ClassificationCounterReducer extends TableReducer<IntWritable,IntWritable,Put> {

此外，取消设置 Combiner，它不会工作。