hadoop - Hortonworks HA Namenodes 给出错误 "Operation category READ is not supported in state standby"

Question

我的 hadoop 集群 HA 事件名称节点 (host1) 突然切换到备用名称节点 (host2)。我无法在 hadoop 日志(在任何服务器中)中找到任何错误来确定根本原因。

切换名称节点后，hdfs 日志中经常出现以下错误，并且应用程序无法读取 HDFS 文件。

2014-07-17 01:58:53,381 WARN namenode.FSNamesystem (FSNamesystem.java:getCorruptFiles(6769)) - Get corrupt file blocks returned error: Operation category READ is not supported in state standby

一旦我重新启动新的事件节点(host2)，namenode 就会切换回新的备用节点(host1)。然后集群正常工作，用户也可以检索HDFS文件。

我正在使用 Hortonworks 2.1.2.0 和 HDFS 版本 2.4.0.2.1

编辑:2014 年 7 月 21 日主备namenode切换时在active namenode日志中发现如下日志

NT_SETTINGS-1675610.csv dst=null perm=null 2014-07-20 09:06:44,746 INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(7755)) - allowed=true
ugi=storm (auth:SIMPLE) ip=/10.0.1.50 cmd=getfileinfo src=/user/tungsten/staging/LEAPSET/PRODUCTS/PRODUCTS-138018 6.csv dst=null perm=null 2014-07-20 09:06:44,747 INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(7755)) - allowed=true ugi=storm (auth:SIMPLE) ip=/10.0.1.50
cmd=getfileinfo src=/user/tungsten/staging/LEAPSET/MERCHANT_SETTINGS/MERCHA NT_SETTINGS-1695794.csv dst=null perm=null 2014-07-20 09:06:44,747 INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(7755)) - allowed=true
ugi=storm (auth:SIMPLE) ip=/10.0.1.50 cmd=getfileinfo src=/user/tungsten/staging/LEAPSET/PRODUCTS/PRODUCTS-139954 1.csv dst=null perm=null 2014-07-20 09:06:44,748 INFO namenode.FSNamesystem (FSNamesystem.java:stopActiveServices(1095)) - Stopping services started for active state 2014-07-20 09:06:44,750 INFO namenode.FSEditLog (FSEditLog.java:endCurrentLogSegment(1153)) - Ending log segment 842249 2014-07-20 09:06:44,752 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(673)) - Number of transactions: 2 Total time for transactions(ms): 0 Number of transactions batched in Syncs: 0 Number of syncs: 1 SyncTimes(ms): 4 35 2014-07-20 09:06:44,774 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(673)) - Number of transactions: 2 Total time for transactions(ms): 0 Number of transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 24 37 2014-07-20 09:06:44,805 INFO namenode.FSNamesystem (FSNamesystem.java:run(4362)) - NameNodeEditLogRoller was interrupted, exiting 2014-07-20 09:06:44,824 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(130)) - Finalizing edits file /ebs/hadoop/hdfs/namenode/current/edits_inprogress_0000000000000842249 -> /ebs/hadoop/hdfs/name node/current/edits_0000000000000842249-0000000000000842250 2014-07-20 09:06:44,874 INFO blockmanagement.CacheReplicationMonitor (CacheReplicationMonitor.java:run(168)) - Shutting down CacheReplicationMonitor 2014-07-20 09:06:44,876 INFO namenode.FSNamesystem (FSNamesystem.java:startStandbyServices(1136)) - Starting services required for standby state 2014-07-20 09:06:44,927 INFO ha.EditLogTailer (EditLogTailer.java:(117)) - Will roll logs on active node at hadoop-client-us-west-1b/10.0.254.10:8020 every 120 seconds. 2014-07-20 09:06:44,929 INFO ha.StandbyCheckpointer (StandbyCheckpointer.java:start(129)) - Starting standby checkpoint thread... Checkpointing active NN at http:// hadoop-client-us-west-1b:50070 Serving checkpoints at http:// hadoop-client-us-west-1a:50070 2014-07-20 09:06:44,930 INFO ipc.Server (Server.java:run(2027)) - IPC Server handler 3 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from 10.0.1.50:57297 Call#8431877 Retry#0: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby 2014-07-20 09:06:44,930 INFO ipc.Server (Server.java:run(2027)) - IPC Server handler 16 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from 10.0.1.50:57294 Call#130105071 Retry#0: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby 2014-07-20 09:06:44,940 INFO ipc.Server (Server.java:run(2027)) - IPC Server handler 14 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from 10.0.1.50:57294 Call#130105072 Retry#0: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby

编辑:2014 年 8 月 13 日我们能够找出 namenode 切换的根本原因，namenode 收到大量文件信息请求，然后发生了 namenode 切换。

但还是无法解决Operation category READ is not supported in state standby错误。

编辑:2014 年 12 月 7 日我们发现，作为解决方案，应用程序需要在先前事件的名称节点失败后手动连接当前事件的名称节点。 HA 模式下名称节点的流量不会自动定向到事件节点。

Answer 1

我遇到了同样的问题。您需要更新客户端库。使用 amabari 设置 spark 并让它在服务器上安装客户端。然后设置您的 SPARK_HOME 环境变量。