cassandra - 停用 cassandra 节点超时，出现 "received only 0 responses"

Question

当我尝试停用我的 Cassandra 集群中的一个节点时，该过程开始(我看到事件流从该节点流向集群中的其他节点(使用 vnodes)停用)，但在稍有延迟之后 nodetool decommission 存在以下错误消息。

我可以重复运行 nodetool decommission，它会开始将数据流式传输到其他节点，但到目前为止始终存在以下错误。

为什么我会看到这个，有什么方法可以安全地停用这个节点吗？

Exception in thread "main" java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
        at org.apache.cassandra.db.HintedHandOffManager.getHintsSlice(HintedHandOffManager.java:578)
        at org.apache.cassandra.db.HintedHandOffManager.listEndpointsPendingHints(HintedHandOffManager.java:528)
        at org.apache.cassandra.service.StorageService.streamHints(StorageService.java:2854)
        at org.apache.cassandra.service.StorageService.unbootstrap(StorageService.java:2834)
        at org.apache.cassandra.service.StorageService.decommission(StorageService.java:2795)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1454)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:74)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1295)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1387)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:818)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
        at sun.rmi.transport.Transport$1.run(Transport.java:159)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
        at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:100)
        at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:1213)
        at org.apache.cassandra.db.HintedHandOffManager.getHintsSlice(HintedHandOffManager.java:573)
        ... 33 more

Answer 1

提示切换管理器正在检查提示以查看它是否需要在期间传递这些提示退役，这样提示就不会丢失。你很可能有很多提示，或者一堆墓碑，或表中的某些内容导致查询超时。你不是在超时之前看到您的日志中的任何其他异常是您吗？提高读取超时在您停用节点或手动删除提示 CF 之前在您的节点上运行一段时间，应该很可能让你过去。如果你删除它们，你会想要确保你完成所有退役后运行完整的集群修复，以传播数据来自您删除的任何提示。