Hadoop Kerberos : Datanode cannot connect to Namenode. 由 jsvc 启动 Datanode 以绑定(bind)特权端口(不使用 SASL)

Question

我设置了一个运行良好的 HA Hadoop 集群。但是添加Kerberos认证后datanode无法连接namenode。

验证 Namenode 服务器成功启动并且没有记录错误。我用用户 'hduser'

启动所有服务

$ sudo netstat -tuplen
...
tcp        0      0 10.28.94.150:8019       0.0.0.0:*               LISTEN      1001       20218      1518/java         
tcp        0      0 10.28.94.150:50070      0.0.0.0:*               LISTEN      1001       20207      1447/java         
tcp        0      0 10.28.94.150:9000       0.0.0.0:*               LISTEN      1001       20235      1447/java         

数据节点

以 root 身份启动 datanode，使用 jsvc 将服务绑定(bind)到特权端口(ref. Secure Datanode )

$ sudo -E sbin/hadoop-daemon.sh start datanode
starting datanode, logging to /opt/hadoop-2.7.3/logs//hadoop-hduser-datanode-STWHDDN01.out

得到datanode无法连接到namenodes的错误:

...
2018-01-08 09:25:40,051 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnUserName = hduser
2018-01-08 09:25:40,052 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = supergroup
2018-01-08 09:25:40,114 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2018-01-08 09:25:40,125 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
2018-01-08 09:25:40,152 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:50020
2018-01-08 09:25:40,219 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: ha-cluster
2018-01-08 09:25:41,189 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: ha-cluster
2018-01-08 09:25:41,226 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2018-01-08 09:25:41,227 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2018-01-08 09:25:42,297 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: STWHDRM02/10.28.94.151:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-01-08 09:25:42,300 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: STWHDRM01/10.28.94.150:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)    

datanode hdfs-site.xml(节选):

<property>
  <name>dfs.block.access.token.enable</name>
  <value>true</value>
</property>
<property>
  <name>dfs.datanode.keytab.file</name>
  <value>/opt/hadoop/etc/hadoop/hdfs.keytab</value>
</property>
<property>
  <name>dfs.datanode.kerberos.principal</name>
  <value>hduser/_HOST@FDATA.COM</value>
</property>
<property>
    <name>dfs.datanode.address</name>
    <value>0.0.0.0:1004</value>
</property>
<property>
    <name>dfs.datanode.http.address</name>
    <value>0.0.0.0:1006</value>
</property>
<property>
    <name>dfs.datanode.data.dir.perm</name>
    <value>700</value>
</property>

我在 hadoop-env.sh 中设置了 HADOOP_SECURE_DN_USER=hduser 和 JSVC_HOME

datanode 上的 hdfs.keytab:

$ klist -ke etc/hadoop/hdfs.keytab                                                             Keytab name: FILE:etc/hadoop/hdfs.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   1 hduser/stwhddn01@FDATA.COM (aes256-cts-hmac-sha1-96)
   1 hduser/stwhddn01@FDATA.COM (aes128-cts-hmac-sha1-96)
   1 hduser/stwhddn01@FDATA.COM (des3-cbc-sha1)
   1 hduser/stwhddn01@FDATA.COM (arcfour-hmac)
   1 hduser/stwhddn01@FDATA.COM (des-hmac-sha1)
   1 hduser/stwhddn01@FDATA.COM (des-cbc-md5)
   1 HTTP/stwhddn01@FDATA.COM (aes256-cts-hmac-sha1-96)
   1 HTTP/stwhddn01@FDATA.COM (aes128-cts-hmac-sha1-96)
   1 HTTP/stwhddn01@FDATA.COM (des3-cbc-sha1)
   1 HTTP/stwhddn01@FDATA.COM (arcfour-hmac)
   1 HTTP/stwhddn01@FDATA.COM (des-hmac-sha1)
   1 HTTP/stwhddn01@FDATA.COM (des-cbc-md5)  

操作系统:Centos 7
Hadoop:2.7.3
Kerberos:MIT 1.5.1

当以 root 用户身份运行 datanode 时，我 guest 它不使用 kerberos 进行身份验证。

有什么想法吗？

Answer 1

我发现了问题。需要更改/etc/hosts 以仅将 127.0.0.1 映射到本地主机。

之前

127.0.0.1 STWHDDD01
127.0.0.1 localhost
...

之后

127.0.0.1 localhost
...

我仍然想知道为什么旧映射在没有 Kerberos 身份验证的情况下工作。