Hadoop Balancer 失败并显示 - IOException : Couldn't set up IO streams (LeaseRenewer Warning)

Question

我在通过 Namenode 运行 Hadoop Balancer 时遇到了这个错误。关于破解这个的任何提示。该进程还会阻止当前用户并在发出任何其他命令时给出内存不足错误。

   14/05/09 11:30:05 WARN hdfs.LeaseRenewer: Failed to renew lease for [DFSClient_NONMAPREDUCE_-77290934_1] for 936 seconds.  Will retry shortly ...

 java.io.IOException: Failed on local exception: java.io.IOException: Couldn't set up IO streams; Host Details : local host is: "hadoop01.xx.xx.xx.xx.com/30.0.1.176"; destination host is: "hadoop01.xx.xx.xx.xx.com":8022; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
        at org.apache.hadoop.ipc.Client.call(Client.java:1242)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
        at $Proxy10.renewLease(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
        at $Proxy10.renewLease(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:458)
        at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:649)
        at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417)
        at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442)
        at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
        at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
        at java.lang.Thread.run(Thread.java:662)
    Caused by: java.io.IOException: Couldn't set up IO streams
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:671)
        at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:252)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1291)
        at org.apache.hadoop.ipc.Client.call(Client.java:1209)
        ... 15 more
    Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:640)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:664)
        ... 18 more

Answer 1

一旦Hadoop RPC创建的线程数达到节点进程数的(ulimit -u)，java将报告内存不足错误。

尝试增加允许的最大进程数，即您的 ulimit -u 值。