python - 我的 boto elastic mapreduce jar 作业流参数有什么问题？

Question

我正在使用 boto 库在 Amazon 的 Elastic MapReduce Web 服务 (EMR) 中创建工作流。以下代码应创建一个步骤:

step2 = JarStep(name='Find similiar items',
            jar='s3n://recommendertest/mahout-core/mahout-core-0.5-SNAPSHOT.jar',
            main_class='org.apache.mahout.cf.taste.hadoop.similarity.item.ItemSimilarityJob',
            step_args=['s3n://bucket/output/' + run_id + '/aggregate_watched/',
                       's3n://bucket/output/' + run_id + '/similiar_items/',
                       'SIMILARITY_PEARSON_CORRELATION'
                      ])

当我运行作业流程时，它总是失败并抛出此错误:

java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/JobContext
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.JobContext

这是 EMR 日志中调用 java 代码的行:

2011-01-24T22:18:54.491Z INFO Executing /usr/lib/jvm/java-6-sun/bin/java \
-cp /home/hadoop/conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop:/home/hadoop \
/hadoop-0.18-core.jar:/home/hadoop/hadoop-0.18-tools.jar:/home/hadoop/lib/*:/home/hadoop/lib/jetty-ext/* \
-Xmx1000m \
-Dhadoop.log.dir=/mnt/var/log/hadoop/steps/3 \
-Dhadoop.log.file=syslog \
-Dhadoop.home.dir=/home/hadoop \
-Dhadoop.id.str=hadoop \
-Dhadoop.root.logger=INFO,DRFA \
-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/3/tmp \
-Djava.library.path=/home/hadoop/lib/native/Linux-i386-32 \
org.apache.hadoop.mapred.JobShell \
/mnt/var/lib/hadoop/steps/3/mahout-core-0.5-SNAPSHOT.jar \
org.apache.mahout.cf.taste.hadoop.similarity.item.ItemSimilarityJob \
s3n://..../output/job_2011-01-24_23:09:29/aggregate_watched/ \
s3n://..../output/job_2011-01-24_23:09:29/similiar_items/ \
SIMILARITY_PEARSON_CORRELATION

参数有什么问题？ Java 类定义可以在这里找到:

https://hudson.apache.org/hudson/job/Mahout-Quality/javadoc/org/apache/mahout/cf/taste/hadoop/similarity/item/ItemSimilarityJob.html

Answer 1

我找到了问题的解决方案:

1. 您需要在作业流参数中指定 hadoop 版本 0.20
2. 您需要使用 mahout-core-0.5-SNAPSHOT-job.jar 运行 JAR 步骤，而不是使用 mahout-core-0.5-SNAPSHOT.jar
3. 如果您的工作流程中有额外的流式处理步骤，则需要修复 boto 中的错误:
   1. 打开boto/emr/step.py
   2. 将第 138 行更改为“return '/home/hadoop/contrib/streaming/hadoop-streaming.jar'”
   3. 保存并重新安装 boto

这是调用 job_flow 函数以与 mahout 一起运行的方式:

jobid = emr_conn.run_jobflow(name = name,
log_uri = 's3n://'+ main_bucket_name +'/emr-logging/',
enable_debugging=1,
hadoop_version='0.20',
steps=[step1,step2])