hadoop - Apache Pig-ORDER BY错误java.lang.ClassCastException:无法将java.lang.String强制转换为java.lang.Integer

Question

我正在尝试在Apache Pig中执行ORDER BY。失败并出现错误java.lang.ClassCastException:无法将java.lang.String强制转换为java.lang.Integer

附件是代码的一部分

  movies = LOAD 'ml-1m/movies.dat' Using PigStorage('\t') as (movieblob:chararray);
A = FOREACH movies GENERATE STRSPLIT (movieblob,'::',3) as movie1:(id:int,title:chararray,genre:chararray);
B = FOREACH A GENERATE movie1.id, movie1.title, TOKENIZE (movie1.genre,'|');
C = FOREACH B GENERATE id, title, FLATTEN ($2) as genre;
C_FLAT = FOREACH C GENERATE FLATTEN ($0) as id:int, FLATTEN ($1) as
title:chararray, FLATTEN ($2) as genre:chararray;
\de C_FLAT 
C_FLAT: {id: int,title: chararray,genre: chararray}
C_FLAT_ORDER = ORDER C_FLAT BY id;
\d C_FLAT_ORDER
2017-01-20 16:17:53,727 [main] ERROR org.apache.pig.tools.grunt.Grunt -   
ERROR 1066: Unable to open iterator for alias C_FLAT_ORDER. Backend error : 
java.lang.ClassCastException: java.lang.String cannot be cast to  
java.lang.Integer

我尝试过并且可行的解决方法是

store C_FLAT INTO 'ml-1m/movies/output' Using PigStorage('|');
m1 = LOAD 'ml-1m/movies/output' Using PigStorage('|') as
(id:int,title:chararray,genre:chararray);
order_m1 = ORDER m1 BY id;
grunt> \de m1
m1: {id: int,title: chararray,genre: chararray}
grunt> \d order_m1 

输出的几行

(3945,Digimon: The Movie (2000),Children's)
(3946,Get Carter (2000),Action)
(3946,Get Carter (2000),Drama)
(3946,Get Carter (2000),Thriller)
(3947,Get Carter (1971),Thriller)
(3948,Meet the Parents (2000),Comedy)
(3949,Requiem for a Dream (2000),Drama)

我努力了

C1 = FOREACH C GENERATE (int)id, title,genre;

并按ID排序，也失败了。任何帮助将不胜感激。

输入文件中使用了一些数据-'ml-1m / movies.dat'

 3945::Digimon: The Movie (2000)::Adventure|Animation|Children's
 3946::Get Carter (2000)::Action|Drama|Thriller
 3947::Get Carter (1971)::Thriller
 3948::Meet the Parents (2000)::Comedy
 3949::Requiem for a Dream (2000)::Drama
 3950::Tigerland (2000)::Drama
 3951::Two Family House (2000)::Drama
 3952::Contender, The (2000)::Drama|Thriller

Answer 1

如果您只是想订购数据，这就是您所需要的

movies = LOAD '/tmp/ml-1m/movies.dat' Using PigStorage('\t') as (movieblob:chararray);
A = FOREACH movies GENERATE FLATTEN(STRSPLIT(movieblob, '::', 3)) AS (id:int,title:chararray,genres:chararray);
B = ORDER A BY id;

我认为您的问题与尝试获取此问题有关

(3946,Get Carter (2000),Action|Drama|Thriller)

入这个

(3946,Get Carter (2000),Action)
(3946,Get Carter (2000),Drama)
(3946,Get Carter (2000),Thriller)

在这种情况下，引用

Splitting a tuple into multiple tuples in Pig

In Apache Pig how can I serialise columns into rows?

Pig Latin split columns to rows

inpt = LOAD '/tmp/ml-1m/movies.dat' Using PigStorage('\t') as (line:chararray);
splt = FOREACH inpt GENERATE FLATTEN(STRSPLIT($0, '::', 3));
movies = FOREACH splt GENERATE (int) $0 as id, (chararray)$1 as title,FLATTEN(STRSPLITTOBAG($2, '\\|', -1)) AS genre;
ordered_movies = ORDER movies BY id;

\d ordered_movies产生

(3945,Digimon: The Movie (2000),Adventure)
(3945,Digimon: The Movie (2000),Animation)
(3945,Digimon: The Movie (2000),Children's)
(3946,Get Carter (2000),Action)
(3946,Get Carter (2000),Drama)
(3946,Get Carter (2000),Thriller)
(3947,Get Carter (1971),Thriller)
(3948,Meet the Parents (2000),Comedy)
(3949,Requiem for a Dream (2000),Drama)
(3950,Tigerland (2000),Drama)
(3951,Two Family House (2000),Drama)
(3952,Contender, The (2000),Drama)
(3952,Contender, The (2000),Thriller)

我仍然觉得Spark Dataframe会更有用。