hadoop - 如何使用加载命令在 pig 的文件夹中加载多个文本文件？

Question

我一直用它来加载一个文本文件

A = LOAD '1try.txt' USING PigStorage(' ') as (c1:chararray,c2:chararray,c3:chararray,c4:chararray);

Answer 1

您可以使用文件夹名代替文件名，如下所示:

A = LOAD 'myfolder' USING PigStorage(' ') 
    AS (c1:chararray,c2:chararray,c3:chararray,c4:chararray);

Pig 将加载指定文件夹中的所有文件，如 Programming Pig 中所述:

When specifying a “file” to read from HDFS, you can specify directories. In this case, Pig will find all files under the directory you specify and use them as input for that load statement. So, if you had a directory input with two datafiles today and yesterday under it, and you specified input as your file to load, Pig will read both today and yesterday as input. If the directory you specify has other directories, files in those directories will be included as well.