hadoop - 在Hadoop中，Namenode如何获取rack的详细信息以及属于某个rack的datanode？

Question

假设在 Hadoop 集群中我们有 2 个机架 rck1 和 rck2。每个机架有 5 个节点。 Namenode如何知道节点1属于机架1，节点3属于机架2。

Answer 1

您必须配置系统以指定机架信息的确定方式。例如，this Cloudera link告诉您如何在 Cloudera Manager 中为主机配置机架。<​​/p>

或者，this Apache link解释了如何通过配置文件在 java 类的外部脚本中指定此信息。

拓扑通常采用/myrack/myhost 形式，但您可以使用更深的层次结构。他们在 python 中有以下示例，它假设每个机架有一个/24 子网，因此提取 IP 地址的前三个字节用作机架号 - 如果您可以相应地设置节点 IP 地址，您可以采用类似的方法，或者编写您自己的脚本来根据每个节点上的 IP 地址或其他可用信息确定机架(即使是主机名和机架之间的简单硬编码映射也可以在您的示例中使用相对较少的节点)。

#!/usr/bin/python
# this script makes assumptions about the physical environment.
#  1) each rack is its own layer 3 network with a /24 subnet, which
# could be typical where each rack has its own
#     switch with uplinks to a central core router.
#
#             +-----------+
#             |core router|
#             +-----------+
#            /             \
#   +-----------+        +-----------+
#   |rack switch|        |rack switch|
#   +-----------+        +-----------+
#   | data node |        | data node |
#   +-----------+        +-----------+
#   | data node |        | data node |
#   +-----------+        +-----------+
#
# 2) topology script gets list of IP's as input, calculates network address, and prints '/network_address/ip'.

import netaddr
import sys
sys.argv.pop(0)                                                  # discard name of topology script from argv list as we just want IP addresses

netmask = '255.255.255.0'                                        # set netmask to what's being used in your environment.  The example uses a /24

for ip in sys.argv:                                              # loop over list of datanode IP's
address = '{0}/{1}'.format(ip, netmask)                      # format address string so it looks like 'ip/netmask' to make netaddr work
try:
   network_address = netaddr.IPNetwork(address).network     # calculate and print network address
   print "/{0}".format(network_address)
except:
   print "/rack-unknown"                                    # print catch-all value if unable to calculate network address