hadoop - PIG Group - 无法获得多个字段

Question

我想知道谁在每个部门领取最高薪水 - 我正在获得每个部门的最高薪水，但无法获得其中的名字....使用 pig 脚本附加文件

EmpData = LOAD '/data/EmpDet3.csv' using PigStorage(',') as 
(fname:chararray,lname:chararray,position:chararray,dept:chararray, salary:chararray);
Grp_Dept = GROUP EmpData by dept;
EmpDataC = FOREACH EmpData GENERATE fname,lname,position,dept,(int)SUBSTRING(salary,1,10) as sal;
Group_Pos = GROUP EmpDataC by position;
Max_Sal = FOREACH Group_Pos GENERATE group,MAX(EmpDataC.sal);
dump Max_Sal;

我越来越像

(FIRE ENGINEER,103331)
(POLICE OFFICER,90778)
(POLICE OFFICER2,86520)
(WATER RATE TAKER,88968)
(CIVIL ENGINEER IV,104736)
(ELECTRICAL MECHANIC,91520)
(ASST TO THE ALDERMAN,70764)
(GENERAL LABORER - DSS,40560)
(CHIEF CONTRACT EXPEDITER,84780)

我还需要下面的详细信息以及人名

(FIRE ENGINEER,Dudolfi,103331)
(POLICE OFFICER,AARON,90778)
(POLICE OFFICER2,ABBATE,86520)
(WATER RATE TAKER,AARON,88968)
(CIVIL ENGINEER IV,ABAD JR,104736)
(ELECTRICAL MECHANIC,ABBATACOLA,91520)
(ASST TO THE ALDERMAN,ABARCA,70764)
(GENERAL LABORER - DSS,ABARCA,40560)
(CHIEF CONTRACT EXPEDITER,AARON,84780)

** 我的输入文件 **

"AARON,  ELVIA J",WATER RATE TAKER,WATER MGMNT,"$88,968.00 "
"AARON,  JEFFERY M",POLICE OFFICER,POLICE,"$80,778.00 "
"AARON,  KARINA",POLICE OFFICER,POLICE,"$90,778.00 "
"AARON,  KIMBERLEI R",CHIEF CONTRACT EXPEDITER,GENERAL SERVICES,"$84,780.00 "
"ABAD JR,  VICENTE M",CIVIL ENGINEER IV,WATER MGMNT,"$104,736.00 "
"ABARCA,  ANABEL",ASST TO THE ALDERMAN,CITY COUNCIL,"$70,764.00 "
"ABARCA,  EMMANUEL",GENERAL LABORER - DSS,STREETS & SAN,"$40,560.00 "
"ABBATACOLA,  ROBERT J",ELECTRICAL MECHANIC,AVIATION,"$91,520.00 "
"ABBATEMARCO,  JAMES J",FIRE ENGINEER,FIRE,"$90,456.00 "
"ABBATE,  TERRY M",POLICE OFFICER2,POLICE,"$86,520.00 "
"XXRON,  KINA",POLICE OFFICER2,POLICE,"$50,778.00 "
"Dudolfi, Cris",FIRE ENGINEER,FIRE,"$103,331.00 "

Answer 1

好的.. 这会起作用的。但还要记住，同一职位下的两个人可以有相同的薪水，如果薪水最高，那么下面的脚本也会生成他们的记录

Emp_Data = LOAD '/data/EmpDet3.csv' using PigStorage(',') as 
(fname:chararray,lname:chararray,position:chararray,dept:chararray, salary:chararray);

Grp_Dept = GROUP Emp_Data by dept;

Emp_DataC = FOREACH Emp_Data GENERATE fname,lname,position,dept,(int)SUBSTRING(salary,1,10) as sal;

Group_Pos = GROUP Emp_DataC by position;

Pos_max_sal = FOREACH Group_Pos GENERATE group as pos,MAX(EmpDataC.sal) as highest_sal;

Emp_max_sal = join  Emp_Data by (position,salary),Pos_max_sal by (pos,highest_sal)

final_set = foreach Emp_max_sal generate position, CONCAT(fname,lname) as emp_name, salary