hadoop - Oozie Hive Action 挂起，心跳永远

Question

我正在尝试通过我在 Hue 中创建的 Oozie 工作流来运行 Hive 操作，但是操作“心跳”永远不会执行 Hive SQL。

我读过其他关于永远心跳的文章，但这篇文章似乎发生在不同的时间点，在 SQL 语句被解析之后。我检查了集群中每个节点的内存，并验证了任务计数参数是否合理。

这是 hive-config.xml 文件:

<configuration>

<property>   
  <name>javax.jdo.option.ConnectionURL</name>   
  <value>jdbc:hive://10.1.10.250:10000/testdb</value>   
  <description>JDBC connect string</description> 
</property>

<property>   
  <name>javax.jdo.option.ConnectionDriverName</name>   
  <value>org.apache.hadoop.hive.jdbc.HiveDriver</value>   
  <description>JDBC driver</description> 
</property>

</configuration>

我知道 Hive 连接正常，因为如果提供错误的 SQL 语句、错误的 URL 或错误的驱动程序名称，操作就会失败。

这是操作标准输出日志:

  [...truncated]

  =================================================================

  >>> Invoking Hive command line now >>>

  4283 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver>
  4284 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>
  4284 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
  4339 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
  4354 [main] INFO  hive.ql.parse.ParseDriver  - Parsing command: create table testdb.temp99 (col1 int)
  4665 [main] INFO  hive.ql.parse.ParseDriver  - Parse Completed
  4667 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=parse start=1418968298270 end=1418968298598 duration=328 from=org.apache.hadoop.hive.ql.Driver>
  4667 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
  4733 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  - Starting Semantic Analysis
  4735 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  - Creating table testdb.temp99 position=13
  4760 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Semantic Analysis Completed
  4775 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=semanticAnalyze start=1418968298598 end=1418968298706 duration=108 from=org.apache.hadoop.hive.ql.Driver>
  4784 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Returning Hive schema: Schema(fieldSchemas:null, properties:null)
  4784 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=compile start=1418968298215 end=1418968298715 duration=500 from=org.apache.hadoop.hive.ql.Driver>
  4785 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Concurrency mode is disabled, not creating a lock manager
  4785 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
  4785 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Starting command: create table testdb.temp99 (col1 int)
  4792 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=TimeToSubmit start=1418968298215 end=1418968298723 duration=508 from=org.apache.hadoop.hive.ql.Driver>
  4792 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
  4792 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver>
  4815 [main] INFO  hive.ql.exec.DDLTask  - Default to LazySimpleSerDe for table testdb.temp99
  4935 [main] INFO  org.apache.hadoop.hive.metastore.HiveMetaStore  - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
  4959 [main] INFO  org.apache.hadoop.hive.metastore.ObjectStore  - ObjectStore, initialize called
  5261 [main] INFO  DataNucleus.Persistence  - Property datanucleus.cache.level2 unknown - will be ignored
  Heart beat
  Heart beat
  [...forever...]

为什么工作流在日志中的这个点跳动而不是继续？

附录:

与此 Hive 操作关联的 Oozie 工作流是:

<workflow-app name="Hive-copy" xmlns="uri:oozie:workflow:0.4">
    <start to="Hive"/>
    <action name="Hive">
        <hive xmlns="uri:oozie:hive-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
              <job-xml>/user/test/hive-config.xml</job-xml>
            <script>/user/test/test.sql</script>
            <file>hive-config.xml#hive-config.xml</file>
        </hive>
        <ok to="end"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

Answer 1

这个问题与nodeManager有关。尝试在 ma​​pred-site.xml 文件中设置以下属性修复问题:

<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>50 </value> 
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>50 </value> 
</property>

类似于这个问题:https://groups.google.com/a/cloudera.org/forum/?fromgroups=#!topic/cdh-user/v0BHtQ0hlBg