hadoop - HDFS伪分布式模式namenodes启动报错

Question

我正在尝试以伪分布式模式在 Mac OS X (Java 7) 上启动 HDFS。我按照在不同地方找到的说明(例如 https://hadoop.apache.org/docs/r1.2.1/single_node_setup.html )创建了一个包含配置文件的目录。我可以在不需要密码的情况下通过 ssh 连接到本地主机。但是当我尝试启动 hdfs 时，我得到以下信息:

$ start-dfs.sh --config ~/hadoop-pseudodistributed
2014-03-12 01:15:14.125 java[84567:1903] Unable to load realm info from SCDynamicStore
Starting namenodes on [2014-03-12 01:15:14,380 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
localhost]
2014-03-12: ssh: Could not resolve hostname 2014-03-12: nodename nor servname provided, or not known
Unable: ssh: Could not resolve hostname Unable: nodename nor servname provided, or not known
[main]: ssh: Could not resolve hostname [main]: nodename nor servname provided, or not known
WARN: ssh: Could not resolve hostname WARN: nodename nor servname provided, or not known
load: ssh: Could not resolve hostname load: nodename nor servname provided, or not known
-: ssh: Could not resolve hostname -: nodename nor servname provided, or not known
for: ssh: Could not resolve hostname for: nodename nor servname provided, or not known
native-hadoop: ssh: Could not resolve hostname native-hadoop: nodename nor servname provided, or not known
where: ssh: Could not resolve hostname where: nodename nor servname provided, or not known
builtin-java: ssh: Could not resolve hostname builtin-java: nodename nor servname provided, or not known
your: ssh: Could not resolve hostname your: nodename nor servname provided, or not known
applicable: ssh: Could not resolve hostname applicable: nodename nor servname provided, or not known
(NativeCodeLoader.java:<clinit>(62)): ssh: Could not resolve hostname (NativeCodeLoader.java:<clinit>(62)): nodename nor servname provided, or not known
using: ssh: Could not resolve hostname using: nodename nor servname provided, or not known
classes: ssh: Could not resolve hostname classes: nodename nor servname provided, or not known
platform...: ssh: Could not resolve hostname platform...: nodename nor servname provided, or not known
library: ssh: Could not resolve hostname library: nodename nor servname provided, or not known
localhost: starting namenode, logging to /usr/local/Cellar/hadoop/2.2.0/libexec/logs/hadoop-terry-namenode-Terrys-MacBook-Pro.local.out
01:15:14,380: ssh: Could not resolve hostname 01:15:14,380: nodename nor servname provided, or not known
to: ssh: connect to host to port 22: Connection refused
localhost: 2014-03-12 01:15:15,150 INFO  [main] namenode.NameNode (StringUtils.java:startupShutdownMessage(601)) - STARTUP_MSG:

有更多的输出(我收到类似的提示，因为它试图启动辅助名称节点)，但上面的内容显然是不可取的，我显然想修复它。

看起来脚本正在运行一些东西来获取名称节点列表，并且那个东西正在转储错误(到标准输出或标准错误)，该错误被捕获并用作节点列表。

我尝试通过添加到 hadoop-env.sh(如 stackoverflow 上其他地方的建议)来清除“无法从 SCDynamicStore 加载领域信息”错误，但未成功。但这似乎超出了记录的设置步骤，其中不包括将 hadoop-env.sh 的副本放入我的配置目录。

我想这应该很容易，但现在已经很晚了，我累了:-(任何帮助将不胜感激。

谢谢!

Answer 1

当我尝试在 MacOS 上以伪分布式模式运行 Hadoop 时，我有一个非常令人满意的解决方法。

# use hadoop-daemon.sh instead of start-dfs.sh
# because start-dfs.sh relies on native libs not present in MacOS
/platform/hadoop/sbin/hadoop-daemon.sh start namenode
/platform/hadoop/sbin/hadoop-daemon.sh start secondarynamenode
/platform/hadoop/sbin/hadoop-daemon.sh start datanode

# use hadoop-daemon.sh instead of stop-dfs.sh
# because stop-dfs.sh relies on native libs not present in MacOS
/platform/hadoop/sbin/hadoop-daemon.sh stop datanode
/platform/hadoop/sbin/hadoop-daemon.sh stop secondarynamenode
/platform/hadoop/sbin/hadoop-daemon.sh stop namenode

我知道这已经 3 年了，但希望这可以让其他人避免我经历的麻烦和浪费的精力——我花了太多时间试图从源代码构建以获得你需要通过更改指向的 native 库到 hadoop-env.sh 使 start-dfs.sh 和 stop-dfs.sh 工作，在看到这个问题之前，阅读脚本以查看它们在调用什么($HADOOP_PREFIX/bin/hdfs getconf -namenodes)并意识到自从我只对伪分布式模式感兴趣，每种类型都有一个节点，我只能说他妈的'方便'脚本并使用 hadoop-daemon.sh 自己启动和停止它们。

而且我个人不需要这个，但如果您正在执行某些脚本被其他东西调用的操作，您甚至可以使用此解决方法覆盖 start-dfs.sh 和 stop-dfs.sh 的内容。