hadoop - apache 配置单元无法连接到 derby :metastore_db although ij from derby can

Question

我正在三台机器上的 hadoop 集群上设置一个 hive。 hadoop (2.7.1) 和 derby (10.11) 运行良好:

hduser@master:~$ ij
ij version 10.11
ij> connect 'jdbc:derby://localhost:1527/metastore_db;create=true';
ij> select * from a;
ID
-----------

0 rows selected
ij>

但是 hive 提示:

...
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
        ... 8 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
        ... 14 more
Caused by: javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the given database. JDBC url = jdbc:derby://localhost:1527/metastore_db;create=true , username = APP. Terminating connectio pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
java.sql.SQLException: No suitable driver found for jdbc:derby://localhost:1527/metastore_db;create=true
...

环境变量:

# Java
export JAVA_HOME=/usr/lib/jvm/jdk

# derby

export DERBY_HOME=/usr/local/derby
export PATH=$PATH:$DERBY_HOME/bin
export CLASSPATH=$CLASSPATH:$DERBY_HOME/lib/derby.jar:$DERBY_HOME/lib/derbytools.jar

# Hadoop
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"


# hive

export HIVE_HOME=/usr/local/hive
export PATH=$PATH:$HIVE_HOME/bin
export CLASSPATH=$CLASSPATH:/usr/local/hive/lib/*:.

hive-site.xml 在配置中只有这个属性:

  <property>
     <name>javax.jdo.option.ConnectionURL</name>
     <value>jdbc:derby://localhost:1527/metastore_db;create=true </value>
     <description>JDBC connect string for a JDBC metastore </description>
  </property>

这是怎么回事？

Answer 1

你说:

export CLASSPATH=$CLASSPATH:$DERBY_HOME/lib/derby.jar:$DERBY_HOME/lib/derbytools.jar

但你还说:

jdbc:derby://localhost:1527/metastore_db;create=true

因此，您为 Derby 的客户端/服务器配置提供了一个 JDBC 连接 URL，但您提供的 CLASSPATH 是为 Derby 的嵌入式配置提供的。

因此您收到错误:

No suitable driver found

您可以更改 CLASSPATH，也可以更改 JDBC 连接 URL。

如果您想使用 Derby 的客户端/服务器配置，请将您的 CLASSPATH 切换为:

export CLASSPATH=$CLASSPATH:$DERBY_HOME/lib/derbyclient.jar

因为此配置需要 JDBC 客户端驱动程序。

如果您想使用 Derby 的嵌入式配置，请不要管您的 CLASSPATH 并将您的 JDBC 连接 URL 更改为:

jdbc:derby:metastore_db;create=true

要了解有关不同 Derby 配置的更多信息，请阅读此处:http://db.apache.org/derby/docs/10.11/getstart/cgsquck70629.html