java - JVM 状态确定为不稳定。强行退出。什么是 Java 稳定性检查器？？为什么要停止 DSE？

Question

从最近几天开始，我在 cassandra 集群的一些节点上看到 DSE 由于以下错误而关闭，我需要终止 Java 进程并重新启动 DSE 服务。

我已经交叉检查了读写和压缩，没有任何可疑之处，但我看到这些服务器上的 Gc 完全暂停就在问题发生之前。这可能是什么原因如何调试？当这些节点像集群的其余部分一样服务相同的请求时，如何细化为什么在这些特定节点上会发生这种情况？？

发生这种情况是因为没有正确执行 Full Gc，我们使用 G1GC 和 DSE 4.8.3

    ERROR [SharedPool-Worker-25] 2016-12-27 10:14:26,100  JVMStabilityInspector.java:117 - JVM state determined to be unstable.  Exiting forcefully due to:java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3181) ~[na:1.8.0_74]
at org.apache.cassandra.db.RangeTombstoneList.copy(RangeTombstoneList.java:112) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.DeletionInfo.copy(DeletionInfo.java:104) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:217) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.Memtable.put(Memtable.java:210) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1230) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_74]
            at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]

    ERROR [SharedPool-Worker-25] 2016-12-27 10:14:28,100  SEPWorker.java:141 - Failed to execute task, unexpected exception killed worker: {}
    java.lang.IllegalStateException: Shutdown in progress
            at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[na:1.8.0_74]
            at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[na:1.8.0_74]
            at org.apache.cassandra.service.StorageService.removeShutdownHook(StorageService.java:764) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:119) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:68) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
            at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:168) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
at 
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
at 
java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]

    INFO  [Thread-6] 2016-12-27 10:14:56,150  DseDaemon.java:420 - DSE shutting down...

Answer 1

检查 Cassandra 表中是否存在任何大分区，以及是否有任何查询正在读取大量墓碑。