java - 如何使用 Kerberos 在 hive UDF 中获得 HBase 连接？

Question

我想编写一个UDF来从HBase获取一些东西，我用它来将 token 设置为hiveconf，但是我无法使用hiveconf连接到HBase，它会抛出NullPointException

我尝试了很多方法，例如:https://www.programcreek.com/java-api-examples/index.php?api=org.apache.hadoop.hbase.security.token.TokenUtil

但它仍然抛出 NullPointException

import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.apache.hadoop.hbase.client.Connection;

import org.apache.hadoop.hbase.client.ConnectionFactory;

import org.apache.hadoop.hbase.security.User;

import org.apache.hadoop.hbase.security.token.AuthenticationTokenIdentifier;

import org.apache.hadoop.hbase.security.token.TokenUtil;

import org.apache.hadoop.hive.conf.HiveConf;

import org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext;

import org.apache.hadoop.hive.ql.hooks.HookContext;

import org.apache.hadoop.security.UserGroupInformation;

import org.apache.hadoop.security.token.Token;

public class HbaseTokenFetcherHook implements ExecuteWithHookContext{   

    private static final Log LOG = LogFactory.getLog(HbaseTokenFetcherHook.class);

    @Override

    public void run(HookContext hookContext) throws Exception {

          HiveConf hiveConf = hookContext.getConf();

          /* If required */

          hiveConf.set(“zookeeper.znode.parent”, "/hbase-secure");   

          try {               

               UserGroupInformation.setConfiguration(hiveConf);

               Connection tokenConnection = ConnectionFactory.createConnection(hiveConf);

               Token<AuthenticationTokenIdentifier> token = TokenUtil.obtainToken(tokenConnection, User.getCurrent());

               String urlString = token.encodeToUrlString();

               hiveConf.set(“HBASE_AUTH_TOKEN”, urlString);

          } catch (IOException | InterruptedException e) {

               LOG.error("Error while fetching token for hbase"

                         + e.getMessage(), e);

          }

     }

}

它抛出异常:

Token<AuthenticationTokenIdentifier> token = TokenUtil.obtainToken(tokenConnection, User.getCurrent());

错误信息:引起原因:java.lang.NullPointException 在 org.apache.hadoop.hbase.zookeeper.ZookeeperWatcher.getMetaReplicaNodes(ZookeeperWatcher.java:497) 在 org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:558)

Answer 1

尝试Propolis第三方Hive UDF，它具有各种HBase get函数来获取值或整个家族。有关如何构建和使用的说明，请参阅自述文件，了解有关特定 UDF 类型describe 和函数名称的更多信息。我在使用 Kerberos 的 Hadoop 集群上进行了测试，效果很好。