security - 用户的Kerberized Hadoop登录失败…LoginException:校验和失败

Question

环境:

Hadoop 2.9.2

Kerberos 5版本1.15.1

RHEL 7

错误

Hadoop数据节点日志中的异常阻止了启动。

日志条目是:

INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: org.apache.hadoop.security.KerberosAuthException: Login failure for user: datanode/_HOST@<REALM> from keytab /etc/security/keytabs/<file.keytab> javax.security.auth.login.LoginException: Checksum failed

全栈跟踪:

org.apache.hadoop.security.KerberosAuthException: Login failure for user: datanode/_HOST@<REALM> from keytab /etc/security/keytabs/datanode.keytab javax.security.auth.login.LoginException: Checksum failed
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1104)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:312)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2596)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2645)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2789)
        at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.start(SecureDataNodeStarter.java:77)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
Caused by: javax.security.auth.login.LoginException: Checksum failed
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:808)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1095)
        ... 10 more
Caused by: KrbException: Checksum failed
        at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:102)
        at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94)
        at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175)
        at sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:150)
        at sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)
        at sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:308)
        at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:447)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:780)
        ... 23 more
Caused by: java.security.GeneralSecurityException: Checksum failed
        at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451)
        at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272)
        at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76)
        at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:100)
        ... 30 more

诊断

如果运行 kdiag 来诊断问题:

bin/hadoop org.apache.hadoop.security.KDiag --principal namenode/_HOST@<REALM> --keytab /etc/security/keytab/namenode.keytab

遇到与上述相同的异常...

有用输出的最后一行是:

>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType

正常的健康输出应为:

>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> CksumType: sun.security.krb5.internal.crypto.HmacSha1Aes256CksumType
>>> KrbAsRep cons in KrbAsReq.getReply datanode/_HOST

/etc/krb5.conf

Kerberos配置的内容

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
  dns_lookup_realm = false
  dns_lookup_kdc = false
  ticket_lifetime = 24h
  renew_lifetime = 7d
  rdns = false
  forwardable = true
  # pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
  default_realm = EXAMPLE.COM
  default_ccache_name = KEYRING:persistent:%{uid}

[realms]
  EXAMPLE.COM = {
  kdc = kdc.example.com
  admin_server = kdc.example.com
  dict_file = /usr/share/dict/words
}
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM

Answer 1

选项1

从renew_lifetime删除krb5.conf
根据this

选项2

确保已设置renew_lifetime，ticket_lifetime和max_renewable_life。

示例工作配置

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
  default_realm = EXAMPLE.COM
  dns_lookup_kdc = false
  dns_lookup_realm = false
  ticket_lifetime = 24h
  renew_lifetime = 7d
  forwardable = true
  default_tgs_enctypes = aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des- cbc-crc
  default_tkt_enctypes = aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc
  permitted_enctypes = aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc
  kdc_timeout = 3000

[realms]
  EXAMPLE.COM = {
  kdc = kdc.example.com
  admin_server = kdc.example.com
  dict_file = /usr/share/dict/words
  max_renewable_life = 7d 0h 0m 0s
  }

[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM