hadoop - pig 的NOT IN功能

Question

我正在尝试使用Pig中的DIFF()方法找出两个表(源表和目标表)之间的差异，以实现以下目标:

sourcenew = LOAD 'hdfs://HADOOPMASTER:54310/DVTTest/Source.txt' USING PigStorage(',') as (ID:chararray,Name:chararray,FirstName:chararray ,LastName:chararray,Vertical_Name:chararray ,Vertical_ID:chararray,Gender:chararray,DOB:chararray,Degree_Percentage:chararray ,Salary:chararray,StateName:chararray);


destnew = LOAD 'hdfs://HADOOPMASTER:54310/DVTTest/Destination.txt' USING PigStorage(',') as (ID:chararray,Name:chararray,FirstName:chararray ,LastName:chararray,Vertical_Name:chararray ,Vertical_ID:chararray,Gender:chararray,DOB:chararray,Degree_Percentage:chararray ,Salary:chararray,StateName:chararray);

cogroupnew= COGROUP sourcenew by ID inner, destnew by ID inner;

diffnew = FOREACH cogroupnew GENERATE DIFF(sourcenew,destnew);

DUMP diffnew;

给出两个表之间的差异，或者如果元组匹配，则返回空包{}，在此之前它可以正常工作，下一步是在源文件中找到目标中没有的额外记录，为此

cogroupextrainsource= COGROUP sourcenew by ID inner, destnew by ID;
filterextrainsource= FILTER cogroupextrainsource BY ID NOT (cogroupnew)

预期的 throw 错误。
需要帮助以找到更多资源。
帮助将不胜感激。

谢谢!

Answer 1

您不需要列名ID旁边的$符号。仅当您不想按名称访问列时，才使用$。

cogroupextrainsource = COGROUP sourcenew by ID inner, destnew by ID;