Hadoop 安全模式恢复 - 花费的时间太长了!

Question

我有一个包含 18 个数据节点的 Hadoop 集群。我在两个多小时前重启了名称节点，名称节点仍处于安全模式。

我一直在寻找为什么这可能会花费太长时间，但找不到好的答案。发帖在这里: Hadoop safemode recovery - taking lot of time是相关的，但我不确定是否要/需要在更改此设置后重新启动名称节点，如该文章所述:

<property>
 <name>dfs.namenode.handler.count</name>
 <value>3</value>
 <final>true</final>
</property>

无论如何，这就是我在“hadoop-hadoop-namenode-hadoop-name-node.log”中得到的:

2011-02-11 01:39:55,226 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call delete(/tmp/hadoop-hadoop/mapred/system, true) from 10.1.206.27:54864: error: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /tmp/hadoop-hadoop/mapred/system. Name node is in safe mode.
The reported blocks 319128 needs additional 7183 blocks to reach the threshold 0.9990 of total blocks 326638. Safe mode will be turned off automatically.
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /tmp/hadoop-hadoop/mapred/system. Name node is in safe mode.
The reported blocks 319128 needs additional 7183 blocks to reach the threshold 0.9990 of total blocks 326638. Safe mode will be turned off automatically.
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1711)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1691)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:565)
    at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:966)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:962)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:416)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:960)

欢迎任何建议。谢谢!

Answer 1

我遇到过一次，其中一些 block 从未报告过。我不得不强行让 namenode 离开安全模式(hadoop dfsadmin -safemode leave)，然后运行 ​​fsck 来删除丢失的文件。