Hadoop: pig 错误

Question

我是 Hadoop 的绝对初学者，我只是在做一些简单的测试，但是，我发现错误消息并没有提供非常有用的信息。

我在 CentOS 6.4 VM 上以单节点模式设置了我的 Hadoop 环境，有 4Gb 可用内存。

我正在尝试在一个 500Mb CSV 文件上运行一个简单的 Pig 脚本。我有两个 500Mb 的文件，在第一个上，脚本成功了。在第二个上，它的大小大致相同，但数据不同(行数更多)，当执行达到大约 60% 时出现错误。

这是我使用的(非常简单的)Pig 脚本:

records = LOAD 'trans2013.csv' USING PigStorage(',') AS
(podracun_v_breme,datum_transakcije,znesek_transakcije,oznaka_valute_transakcije,racun_v_dobro,naziv_prejemnika,maticna_stevilka,davcna_stevilka,sifra_pu,zr_sns_oe,namen);
transaction_recs = GROUP records ALL;
tot_trans = FOREACH transaction_recs GENERATE
SUM(records.znesek_transakcije);
STORE tot_trans INTO '/user/root/totaltransactions';

这是我在终端中得到的错误:

2014-04-06 10:28:29,147 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 64% complete 2014-04-06 10:28:30,240 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure. 2014-04-06 10:28:30,241 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_1396637732046_0008 has failed! Stop running all dependent jobs 2014-04-06 10:28:30,241 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete 2014-04-06 10:28:30,460 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.

2014-04-06 10:28:30,461 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed! 2014-04-06 10:28:30,463 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features 2.0.6-alpha 0.11.1 root 2014-04-06 10:25:49 2014-04-06 10:28:30 GROUP_BY

Failed!

Failed Jobs: JobId Alias Feature Message Outputs job_1396637732046_0008 records,tot_trans,transaction_recs GROUP_BY,COMBINER Message: Job failed! /user/root/totaltransactions,

Input(s): Failed to read data from "hdfs://localhost:8020/user/root/trans2013.csv"

Output(s): Failed to produce result in "/user/root/totaltransactions"

Counters: Total records written : 0 Total bytes written : 0 Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0

Job DAG: job_1396637732046_0008

2014-04-06 10:28:30,463 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed! 2014-04-06 10:28:30,491 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.

Details at logfile: /root/pig_1396797945352.log

这是日志中的错误:

Backend error message --------------------- AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.

Pig Stack Trace --------------- ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.

org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1396637732046_0008_m_000001_0 Info:Container killed by the ApplicationMaster.

at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:217) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:149) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:400) at org.apache.pig.PigServer.launchPlan(PigServer.java:1264) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249) at org.apache.pig.PigServer.execute(PigServer.java:1239) at org.apache.pig.PigServer.executeBatch(PigServer.java:333) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84) at org.apache.pig.Main.run(Main.java:604) at org.apache.pig.Main.main(Main.java:157) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Answer 1

所以...我尝试使用 mapreduce 选项运行 Pig 脚本:

pig -x mapreduce script.pig

它仍然失败了，但它至少产生了一个有意义的错误。看来我不得不从 CSV 文件中删除标题，因为 Pig 使用标题行作为数据。这似乎只在处理 float 时才会发生——如果对整数使用相同的脚本，标题行将被忽略。

原来如此。首先从文件中删除 header ，然后针对它运行脚本 - 它起作用了。