hadoop - HFile 创建 - 添加了一个在词法上不大于前一个键的键

Question

我有一个创建 Put 文件的程序 -

    Put put = new Put(Bytes.add(someKey));
    put.add(COLUMN_FAMILY, colName, timeStamp, dataByteArr); 
    return put;

我正在尝试使用以下代码为这些 Put 创建 Hfile。

    FileInputFormat.setInputPaths(job, new Path(baseDir + "/" + childInputDir + "*"));

    job.setInputFormatClass(TolerantSequenceFileInputFormat.class);

    job.setMapperClass(KeyPutImporter.class);
    HTable htable = new HTable(conf, tableName);

    job.setMapOutputKeyClass(ImmutableBytesWritable.class);
    job.setMapOutputValueClass(Put.class);
    job.setOutputFormatClass(HFileOutputFormat.class);

    HFileOutputFormat.configureIncrementalLoad(job, htable);
    Path hfileOutputPath = new Path(baseDir + "/" + childOutputDir);
    HFileOutputFormat.setOutputPath(job, hfileOutputPath);

    TableMapReduceUtil.addDependencyJars(job.getConfiguration(), com.google.common.base.Preconditions.class);

    boolean success = job.waitForCompletion(true);

在创建 HFile 时，出现以下异常。

    java.io.IOException: Added a key not lexically larger than previous key=
    at org.apache.hadoop.hbase.io.hfile.HFile$Writer.checkKey(HFile.java:577)
    at org.apache.hadoop.hbase.io.hfile.HFile$Writer.append(HFile.java:533)
    at org.apache.hadoop.hbase.io.hfile.HFile$Writer.append(HFile.java:501)
    at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:141)
    at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:96)
    at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:514)
    at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
    at org.apache.hadoop.hbase.mapreduce.PutSortReducer.reduce(PutSortReducer.java:72)

你能帮忙吗？

Answer 1

使用批量加载器(LoadIncrementalHFiles、doBulkLoad)时，您只能添加“按词法排序”的项目，即。您需要确保您添加的项目按行 ID 排序。