hadoop - CDH5 hdfs 平衡器错误

Question

运行“hdfs balancer”，我确实看到它显示了安装的机架拓扑、过度使用和未充分使用的节点，它说“需要移动 5 TB 以使集群平衡”和“决定从 10.150 移动 10 GB 字节.11.24:50010 到 10.150.11.164:50010"

然后我看到这个错误，

WARN balancer.Balancer: Dispatcher thread failed java.lang.NullPointerException at org.apache.hadoop.hdfs.server.balancer.Balancer.isGoodBlockCandidate(Balancer.java:1233) at org.apache.hadoop.hdfs.server.balancer.Balancer.access$400(Balancer.java:183) at org.apache.hadoop.hdfs.server.balancer.Balancer$Source.isGoodBlockCandidate(Balancer.java:686) at org.apache.hadoop.hdfs.server.balancer.Balancer$Source.getBlockList(Balancer.java:674) at org.apache.hadoop.hdfs.server.balancer.Balancer$Source.dispatchBlocks(Balancer.java:776) at org.apache.hadoop.hdfs.server.balancer.Balancer$Source.access$1600(Balancer.java:607) at org.apache.hadoop.hdfs.server.balancer.Balancer$Source$BlockMoveDispatcher.run(Balancer.java:614) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)

然后这个，

No block has been moved for 5 iterations. Exiting...

WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /system/balancer.id: File does not exist. Holder DFSClient_NONMAPREDUCE_-201468433_1 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2937) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2757) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2665) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:569) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)

namenode 日志会显示这一点，

WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:SIMPLE) cause:org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /system/balancer.id: File does not exist. Holder DFSClient_NONMAPREDUCE_-201468433_1 does not have any open files.

INFO org.apache.hadoop.ipc.Server: IPC Server handler 118 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 10.150.11.126:56490 Call#71 Retry#0: error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /system/balancer.id: File does not exist. Holder DFSClient_NONMAPREDUCE_-201468433_1 does not have any open files.

我搜索并发现了一些关于重复平衡器进程的讨论，但我在我们的集群中找不到这个问题。任何人有其他想法？我们正在使用 cdh5.0.1

Answer 1

在 CDH 中，您应该有一个运行 HDFS 平衡器服务的节点。这会阻止命令行中的 hdfs 平衡器。

看这里:

http://community.cloudera.com/t5/Storage-Random-Access-HDFS/Running-HDFS-Balancer-fail-report-namenode-LeaseExpiredException/td-p/14052