hadoop - 以编程方式创建 HFile 并将其加载到 HBase 时，新条目不可用

Question

我正在尝试以编程方式创建 HFile 并将它们加载到正在运行的 HBase 实例中。我在 HFileOutputFormat 和 LoadIncrementalHFiles

中找到了很多信息

我设法创建了新的 HFile，并将其发送到集群。在集群 Web 界面中，新的存储文件出现，但新的键范围不可用。

InputStream stream = ProgrammaticHFileGeneration.class.getResourceAsStream("ga-hourly.txt");
BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
String line = null;

Map<byte[], String> rowValues = new HashMap<byte[], String>();

while((line = reader.readLine())!=null) {
    String[] vals = line.split(",");
    String row = new StringBuilder(vals[0]).append(".").append(vals[1]).append(".").append(vals[2]).append(".").append(vals[3]).toString();
    rowValues.put(row.getBytes(), line);
}

List<byte[]> keys = new ArrayList<byte[]>(rowValues.keySet());
Collections.sort(keys, byteArrComparator);


HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster();

testingUtility.createTable("table".getBytes(), "data".getBytes());

Writer writer = new HFile.Writer(testingUtility.getTestFileSystem(),
    new Path("/tmp/hfiles/data/hfile"),
    HFile.DEFAULT_BLOCKSIZE, Compression.Algorithm.NONE, KeyValue.KEY_COMPARATOR);

for(byte[] key:keys) {
    writer.append(new KeyValue(key, "data".getBytes(), "d".getBytes(), rowValues.get(key).getBytes()));
}

writer.appendFileInfo(StoreFile.BULKLOAD_TIME_KEY, Bytes.toBytes(System.currentTimeMillis()));
writer.appendFileInfo(StoreFile.MAJOR_COMPACTION_KEY, Bytes.toBytes(true));
writer.close();

Configuration conf = testingUtility.getConfiguration();

LoadIncrementalHFiles loadTool = new LoadIncrementalHFiles(conf);
HTable hTable = new HTable(conf, "table".getBytes());

loadTool.doBulkLoad(new Path("/tmp/hfiles"), hTable);

ResultScanner scanner = hTable.getScanner("data".getBytes());
Result next = null;
System.out.println("Scanning");
while((next = scanner.next()) != null) {
    System.out.format("%s %s\n", new String(next.getRow()), new String(next.getValue("data".getBytes(), "d".getBytes())));
}

有人真的做了这个工作吗？我在 my github 上有一个可编译/可测试的版本

Answer 1

查看 hbase 源代码中的 LoadIncrementalHFiles 测试:https://github.com/apache/hbase/blob/7c46646994b7a9d6f947cf12796579ef48d0b0bd/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java