java - Hive 元存储遭受 kerberos "Clock skew too great"错误

Question

最近我们每个月都会遇到一次如标题所述的问题。在metastore节点上，我们安装并启动了ntpd服务以与kerberos服务器同步时间。节点上的 krb5.conf 如下所示:

[lib默认值]
default_realm = Examples.COM
dns_lookup_realm = true
dns_lookup_kdc = true
Ticket_lifetime = 24小时
renew_lifetime = 7d
可转发= true

因此，Metastore 上的时间与 kerberos 服务器不同步(>=5 分钟)似乎不太可能导致问题或由于网络阻塞。
从Metastore日志中看到，“Clock skew Too Great”异常记录时间是乱序的，如，

2016-01-16 18:18:48,071 错误 [pool-3-thread-63735]
2016-01-16 19:07:03,699 错误 [pool-3-thread-63798]
2016-01-16 19:06:55,998 错误 [pool-3-thread-63796]
2016-01-16 19:06:41,653 错误 [pool-3-thread-63812]
2016-01-16 19:04:28,659 错误 [pool-3-thread-63806]
2016-01-16 19:04:13,937 错误 [pool-3-thread-63804]
2016-01-16 19:02:19,312 错误 [pool-3-thread-63809]
2016-01-16 19:02:13,115 错误 [pool-3-thread-63794]
2016-01-16 19:02:06,028 错误 [pool-3-thread-63800]
2016-01-16 19:01:50,767 错误 [pool-3-thread-63795]
2016-01-16 18:59:36,926 错误 [pool-3-thread-63810]
2016-01-16 18:59:36,394 错误 [pool-3-thread-63797]

异常堆栈:

2016-01-16 18:59:36,394 ERROR [pool-3-thread-63797]: transport.TSaslTransport (TSaslTransport.java:open(296)) - SASL negotiation failurejavax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Clock skew too great (37))]        at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)        at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:509)        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:264)        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$HiveSaslServerTransportFactory.getTransport(HadoopThriftAuthBridge.java:172)        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:678)        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:675)        at java.security.AccessController.doPrivileged(Native Method)        at javax.security.auth.Subject.doAs(Subject.java:356)        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1536)        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:675)        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)        at java.lang.Thread.run(Thread.java:744)Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Clock skew too great (37))        at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$HiveSaslServerTransportFactory.getTransport(HadoopThriftAuthBridge.java:172)          ... 10 more

环境:

 java version "1.7.0_45" Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode) hive-0.13.1.2.1.10.0-hdp

那么如果我想找出根本原因该怎么办呢？有什么建议么？非常感谢。

Answer 1

我也看到过这个错误，就我而言，根本原因与 Kerberos 无关。如果您使用 MySql 数据库作为数据存储，则会出现相当严重的内存泄漏，https://issues.apache.org/jira/browse/HIVE-15551 ，这是在 0.13 中引入的，直到 Hive 1.3.0 才修复。基本上，最初编写代码的人要么忘记，要么没有意识到您必须显式关闭 JDBC 语句，这会导致当您的进程达到其内存限制时出现过多的垃圾收集。一旦发生这种情况，过程中的一切都会逐渐变慢，直到您开始看到这些时钟偏差错误。

您可以通过在 Metastore 进程上运行 jmap 实时直方图来判断这是否是您的问题。如果您看到 JDBC 对象位于列表顶部(在我的例子中是 com.mysql.jdbc.JDBC42ResultSet 和 com.mysql.jdbc.StatementImpl)，那么您可能会遇到此问题。我建议您应用补丁、升级到 Hive 1.3.0，或者使用问题中提到的解决方法来查看是否可以解决问题。