hadoop - 使用哪种 FileInputFormat 读取 Hadoop 存档文件 (HAR) 文件

Question

我使用命令行实用程序创建了一个 har 文件:hadoop archive。

如何在 mapreduce 或 spark 中读取 HAR 文件的内容？是否有可以理解 HAR 文件的 FileInputFormat？

---

按照答案...这里是简单的 pig-script 以防其他人感兴趣:

A =     LOAD 'har:///user/me/my.har/*.parquet'  
        USING parquet.pig.ParquetLoader 
        ('key:chararray')
        ;

Answer 1

来自 Hadoop Archives and MapReduce

Using Hadoop Archives in MapReduce is as easy as specifying a different input filesystem than the default file system. If you have a hadoop archive stored in HDFS in /user/zoo/foo.har then for using this archive for MapReduce input, all you need to specify the input directory as har:///user/zoo/foo.har. Since Hadoop Archives is exposed as a file system MapReduce will be able to use all the logical input files in Hadoop Archives as input.

因此，您应该能够使用您将使用的任何 FileInputFormat 来读取相同文件的 HDFS 目录。