hadoop - 是否可以保存 Hive UDTF 生成的值？

Question

我创建了 Hive 自定义 UDTF。我的新 UDTF 根据输入的 2 列返回新的 3 列。

但是，我无法使用生成的值进行任何 INSERT 操作。例如，

• INSERT OVERWRITE DIRECTORY 'generated_data.csv' SELECT udtf(one, two) FROM table_orig;
• INSERT OVERWRITE TABLE test_table SELECT udtf(one, two) FROM table_orig;

INSERT 查询都返回 NullPointerException，如下所示:

2017-05-30T08:02:45,209 ERROR [main([])]: exec.Task (:()) - Failed to execute tez graph.
java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:430)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:451)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:396)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:134)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

欢迎任何建议，谢谢!

Answer 1

在UDTF之后添加列列表:

INSERT OVERWRITE TABLE test_table SELECT udtf(one, two) as (col1, col2, col3) FROM table_orig;