hadoop - Azkaban - HIVe 作业失败

Question

我正在尝试在 azkaban 中运行 hive 作业

我能够上传 Hive 作业，但 Hive 作业未正确执行。

ERROR [hive-demo] [Azkaban] Failed to build job executor for job hive-demoJob type     'hive' is unrecognized. Could not construct job[{hive.query.01: drop table words;, hive.query.03: describe words;, hive.query.02: create table words (freq int, word string) row format delimited fields terminated by ' ' stored as textfile;, working.dir: /home/hduser/Technology/azkaban/azkaban-executor/executions/44, azk.hive.action: execute.query, azkaban.job.attempt: 0, type: hive, hive.query.05: select * from words limit   10;, hive.query.04: load data local inpath "res/input" into table words;, user.to.proxy:   azkaban, hive.query.06: select freq, count(1) as f2 from words group by freq sort by f2 desc  limit 10;, parent = {azkaban.flow.flowid: hive-demo, azkaban.flow.execid: 44,  azkaban.flow.start.timezone: America/Los_Angeles, azkaban.flow.start.hour: 22,  azkaban.flow.start.second: 17, azkaban.flow.start.year: 2014,  azkaban.flow.start.milliseconds: 111, azkaban.flow.start.minute: 54,  azkaban.flow.start.timestamp: 2014-01-31T22:54:17.111-08:00, azkaban.flow.start.month: 01,  azkaban.flow.projectversion: 2, azkaban.flow.projectid: 4, azkaban.flow.uuid: 29a46293-cc95- 4c66-b2b1-51014205ce3b, azkaban.flow.start.day: 31, parent = {}}}] of type[hive]. 2014/02/01  12:24:17.295 +0530 ERROR [hive-demo] [Azkaban] Failed to build job type, skipping this job    2014/02/01 12:24:17.295 +0530 ERROR [hive-demo] [Azkaban] Job run failed! 2014/02/01 12:24:17.295 +0530 INFO [hive-demo] [Azkaban] Finishing job hive-demo at 1391237657295  2014/02/01 12:24:17.382 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning recently finished  2014/02/01 12:24:17.383 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning execution 43 from  recently finished flows list. 2014/02/01 12:24:17.408 +0530 INFO [hive-demo] [Azkaban] Job  Finished hive-demo with status FAILED 2014/02/01 12:24:17.463 +0530 INFO [hive-demo]  [Azkaban] Finishing up flow. Awaiting Termination 2014/02/01 12:24:17.463 +0530 INFO [hive- demo] [Azkaban] Setting flow status to Failed. 2014/02/01 12:24:17.464 +0530 INFO [hive- demo] [Azkaban] Flow is set to FAILED 2014/02/01 12:24:17.464 +0530 INFO [hive-demo]  [Azkaban] Setting end time for flow 44 to 1391237657464 2014/02/01 12:24:17.552 +0530 INFO  [FlowRunnerManager] [Azkaban] Flow 44 is finished. Adding it to recently finished flows  list. 2014/02/01 12:26:17.383 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning recently  finished 2014/02/01 12:26:17.384 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning execution  44 from recently finished flows list.
 ========================================================================

这是我的plugin.properties中的内容

job.class=azkaban.jobtype.hiveutils.azkaban.HiveViaAzkaban

hive.home=HIVE_HOME

hive.aux.jars.path=/usr/lib/hive/lib
env.HIVE_HOME=$HIVE_HOME
env.HIVE_AUX_JARS_PATH=/usr/lib/hive/lib

#hive.jvm.args=-Dhive.querylog.location=. -Dhive.exec.scratchdir=/tmp/hive-    ${user.to.proxy} -Dhive.aux.jars.path=${hive.aux.jars.path}
jobtype.global.jvm.args=${hive.jvm.args}

这是我的private.properties jobtype.classpath=$HADOOP_HOME/conf,$HADOOP_HOME/lib/*,$HIVE_HOME/lib/*,$HIVE_HOME/conf,/​​usr/lib/hive/lib jobtype.class=azkaban.jobtype.HadoopJavaJob

hive.aux.jars.path=/usr/lib/hive/lib
env.HADOOP_HOME=$HADOOP_HOME
env.HIVE_HOME=$HIVE_HOME
env.HIVE_AUX_JARS_PATH=/usr/lib/hive/lib

hive.home=$HIVE_HOME

jobtype.global.classpath=${hive.classpath.items}
  hive.classpath.items=./*,${hive.home}/lib/*,${hive.home}/conf/,${hive.aux.jars.path}/*,${had    oop.home}/*,${hadoop.home}/lib/*,${hadoop.home}/conf/

我只在执行器文件夹中添加了插件文件夹。

为什么没有检测作业类型。请帮忙。

Answer 1

很难判断发生了什么，因为您既不共享作业本身也不共享日志，但请尝试以下操作并让我们知道这是否有效。创建一个包含以下内容的文件 test.job:

type=hive
azk.hive.action=execute.query
hive.query=CREATE TABLE test(id INT, name STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';
user.to.proxy=hadoop

并按照“creating flow”中的描述提交' 部分。有关 Azkaban (2.1) 开箱即用支持的不同类型工作的更多详细信息，请参阅 job types/plugin documentation .