hive - Flume-HiveSink : java. lang.NoClassDefFoundError:org/apache/hive/hcatalog/streaming/RecordWriter

Question

我正在尝试在我的flume配置中创建一个hive接收器，当我运行flume-ng时，我遇到了一些依赖性问题，如下所示。你能告诉我如何克服这种依赖性吗？看起来存在一些运行时依赖性。我已经正确安装了配置单元，并进行了所需的环境变量设置以指向 HIVE_HOME。任何帮助表示赞赏。谢谢。

2016-01-15 14:41:37,757 (conf-file-poller-0) 
[INFO -org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)]  
Creating instance of sink: hiveSink, type: hive
2016-01-15 14:41:37,763 (conf-file-poller-0) 
[ERROR -      org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunn    able.run(PollingPropertiesFileConfigurationProvider.java:145)] 
Failed to  start  agent because dependencies were not found in classpath. Error follows.

java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/streaming/RecordWriter
at org.apache.flume.sink.hive.HiveSink.createSerializer(HiveSink.java:223)
at org.apache.flume.sink.hive.HiveSink.configure(HiveSink.java:203)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:413)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:98)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.streaming.RecordWriter
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more

Answer 1

我正在 CDH 5.7 发行版上学习 Flume。我确实遇到了同样的问题，并且 Flume-env.sh 脚本似乎对我不起作用，所以我使用 --classpath 命令参数来引用与 hive 和 hive 相关的所有库 -目录。采用反复试验的方法来添加多个 lib 文件夹引用。没有找到太多关于这个论点的文档。

例如flume-ng agent --conf /home/cloudera/flume/ --conf-file /home/cloudera/flume/netcat_memchannel_hivesink.conf --name agent1 --classpath "/usr/lib/hive-hcatalog/share/hcatalog/*":"/usr/lib/hive/lib/*"