hadoop - 使用SORT时Apache Pig拉丁语法错误

Question

程序员你好吗？

我正在编程并运行pig命令，在代码中返回了语法错误，无法解决。

下面，我将发布代码和引发的错误。

显然该错误发生在sortedResults的第41行的语法中

码:

--Load files into relations
month1 = LOAD 'hdfs:/data/big/data/weather/201201hourly.txt' USING PigStorage(',');
month2 = LOAD 'hdfs:/data/big/data/weather/201202hourly.txt' USING PigStorage(',');
month3 = LOAD 'hdfs:/data/big/data/weather/201203hourly.txt' USING PigStorage(',');
month4 = LOAD 'hdfs:/data/big/data/weather/201204hourly.txt' USING PigStorage(',');
month5 = LOAD 'hdfs:/data/big/data/weather/201205hourly.txt' USING PigStorage(',');
month6 = LOAD 'hdfs:/data/big/data/weather/201206hourly.txt' USING PigStorage(',');

--Combine relations
months = UNION month1, month2, month3, month4, month5, month6;

/* Splitting relations
SPLIT months INTO 
        splitMonth1 IF SUBSTRING(date, 4, 6) == '01',
        splitMonth2 IF SUBSTRING(date, 4, 6) == '02',
        splitMonth3 IF SUBSTRING(date, 4, 6) == '03',
        splitRest IF (SUBSTRING(date, 4, 6) == '04' OR SUBSTRING(date, 4, 6) == '04');
*/

/*  Joining relations

stations = LOAD 'hdfs:/data/big/data/QCLCD201211/stations.txt' USING PigStorage() AS (id:int, name:chararray)

JOIN months BY wban, stations by id;

*/

--filter out unwanted data
clearWeather = FILTER months BY SkyCondition == 'CLR';

--Transform and shape relation
shapedWeather = FOREACH clearWeather GENERATE date, SUBSTRING(date, 0, 4) as year, SUBSTRING(date, 4, 6) as month, SUBSTRING(date, 6, 8) as day, skyCondition, dryTemp;

--Group relation specifying number of reducers
groupedMonthDay = GROUP shapedWeather BY (month, day) PARALLEL 10;

--Aggregate relation
aggedResults = FOREACH groupedByMonthDay GENERATE group as MonthDay, AVG(shapedWeather.dryTemp), MIN(shapedWeather.dryTemp), MAX(shapedWeather.dryTemp), COUNT(shapedWeather.dryTemp) PARALLEL 10;

--Sort relation
sortedResults = SORT aggedResults BY $1 DESC;

--Store results in HDFS
STORE sortedResults INTO 'hdfs:/data/big/data/weather/pigresults' USING PigStorage(':');

这是我可以播放代码中包含的文件的返回。

错误:

14/10/20 22:30:47 WARN pig.Main: Cannot write to log file: //pig_1413851447162.log
2014-10-20 22:30:47,165 [main] INFO  org.apache.pig.Main - Apache Pig version 0.12.1 (r1585011) compiled Apr 05 2014, 01:41:34
2014-10-20 22:30:47,486 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /home/eduardo/.pigbootup not found
2014-10-20 22:30:47,510 [main] WARN  org.apache.pig.Main - Cannot write to log file: ///weather.pig1413851447510.log
2014-10-20 22:30:47,629 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://localhost:54310
2014-10-20 22:30:47,770 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: localhost:54311
2014-10-20 22:30:48,005 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. <file home/eduardo/Documentos/pig/weather.pig, line 41, column 0>  Syntax error, unexpected symbol at or near 'sortedResults'
Failed to parse: <file home/eduardo/Documentos/pig/weather.pig, line 41, column 0>  Syntax error, unexpected symbol at or near 'sortedResults'
    at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:241)
    at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:179)
    at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1678)
    at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1411)
    at org.apache.pig.PigServer.parseAndBuild(PigServer.java:344)
    at org.apache.pig.PigServer.executeBatch(PigServer.java:369)
    at org.apache.pig.PigServer.executeBatch(PigServer.java:355)
    at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
    at org.apache.pig.Main.run(Main.java:607)
    at org.apache.pig.Main.main(Main.java:156)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
2014-10-20 22:30:48,008 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <file home/eduardo/Documentos/pig/weather.pig, line 41, column 0>  Syntax error, unexpected symbol at or near 'sortedResults'
2014-10-20 22:30:48,008 [main] WARN  org.apache.pig.tools.grunt.Grunt - There is no log file to write to.
2014-10-20 22:30:48,008 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. <file home/eduardo/Documentos/pig/weather.pig, line 41, column 0>  Syntax error, unexpected symbol at or near 'sortedResults'
    at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1691)
    at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1411)
    at org.apache.pig.PigServer.parseAndBuild(PigServer.java:344)
    at org.apache.pig.PigServer.executeBatch(PigServer.java:369)
    at org.apache.pig.PigServer.executeBatch(PigServer.java:355)
    at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
    at org.apache.pig.Main.run(Main.java:607)
    at org.apache.pig.Main.main(Main.java:156)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: Failed to parse: <file home/eduardo/Documentos/pig/weather.pig, line 41, column 0>  Syntax error, unexpected symbol at or near 'sortedResults'
    at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:241)
    at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:179)
    at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1678)
    ... 15 more

Answer 1

Pig没有SORT命令。您必须尝试使用​​“ORDER”命令。这样尝试

sortedResults = ORDER aggedResults BY $1 DESC;