Hadoop 集群 - "hadoop"用户 ssh 通信

Question

我在 EC2 服务器上设置 Hadoop 2.7.3 集群 - 1 个 NameNode、1 个辅助 NameNode 和 2 个 DataNode。

Hadoop 核心使用 SSH 与从节点通信以启动从节点上的进程。

我们是否需要在所有节点上为 hadoop 用户提供相同的 SSH key ？

将 NameNode 复制或添加到从节点 SSH 凭据的最佳实践/理想方法是什么？

Answer 1

Do we need to have same SSH keys on all the nodes for the hadoop user?

所有节点上都需要相同的公钥

What is the best practice/ideal way to copy or add the NameNode to Slave nodes SSH credentials?

Per documentation:

Namenode:无密码 SSH

Password-less SSH between the name nodes and the data nodes. Let us create a public-private key pair for this purpose on the namenode.

namenode> ssh-keygen

Use the default (/home/ubuntu/.ssh/id_rsa) for the key location and hit enter for an empty passphrase.

Datanodes:设置公钥

The public key is saved in /home/ubuntu/.ssh/id_rsa.pub. We need to copy this file from the namenode to each data node and append the contents to /home/ubuntu/.ssh/authorized_keys on each data node.

datanode1> cat id_rsa.pub >> ~/.ssh/authorized_keys
datanode2> cat id_rsa.pub >> ~/.ssh/authorized_keys
datanode3> cat id_rsa.pub >> ~/.ssh/authorized_keys

Namenode:设置 SSH 配置

SSH uses a configuration file located at ~/.ssh/config for various parameters. Set it up as shown below. Again, substitute each node’s Public DNS for the HostName parameter (for example, replace with EC2 Public DNS for NameNode).

Host nnode
  HostName <nnode>
  User ubuntu
  IdentityFile ~/.ssh/id_rsa

Host dnode1
  HostName <dnode1>
  User ubuntu
  IdentityFile ~/.ssh/id_rsa

Host dnode2
  HostName <dnode2>
  User ubuntu
  IdentityFile ~/.ssh/id_rsa

Host dnode3
  HostName <dnode3>
  User ubuntu
  IdentityFile ~/.ssh/id_rsa

At this point, verify that password-less operation works on each node as follows (the first time, you will get a warning that the host is unknown and whether you want to connect to it. Type yes and hit enter. This step is needed once only):

namenode> ssh nnode
namenode> ssh dnode1
namenode> ssh dnode2
namenode> ssh dnode3