java - Apache Ignite OutOfMemory 低使用率

Question

我在多个测试 Kubernetes 集群中部署了 Apache Ignite 的 StatefulSet。

我已经使用当前配置通过了压力测试阶段。但是，我发现 Apache Ignite 中一些负载低得多的新测试集群中存在一些 OutOfMemory 错误。

下面是我从 Apache Ignite 实例中提取的日志快照:

INFO: TCP discovery spawning a new thread for connection [rmtAddr=/10.254.174.226, rmtPort=45453]
Nov 15, 2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: GridWorker [name=tcp-disco-client-message-worker, igniteInstanceName=null, finished=false, heartbeatTs=1573779638619, hashCode=373238347, interrupted=true, runner=tcp-disco-client-message-worker-#109]
java.lang.OutOfMemoryError: Java heap space

Nov 15, 2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: IgniteSpiThread [name=tcp-disco-client-message-worker-#109]
java.lang.OutOfMemoryError: Java heap space
Exception in thread "tcp-disco-client-message-worker-#109" java.lang.OutOfMemoryError: Java heap space
Nov 15, 2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger info
INFO: TCP discovery accepted incoming connection [rmtAddr=/10.254.183.232, rmtPort=41313]
Nov 15, 2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger info
INFO: TCP discovery spawning a new thread for connection [rmtAddr=/10.254.183.232, rmtPort=41313]
Nov 15, 2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Started serving remote node connection [rmtAddr=/10.254.174.226:45453, rmtPort=45453]
Nov 15, 2019 @ 09:01:26.612 Nov 15, 2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger warning
Nov 15, 2019 @ 09:01:26.612 WARNING: New next node has connection to it's previous, trying previous again. [next=TcpDiscoveryNode [id=5cbb5f1c-ca74-4b2f-ba70-314f621ab997, addrs=[10.254.168.12, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, ignite-sit-5.ignite-sit.sit.svc.cluster.local/10.254.168.12:47500], discPort=47500, order=3922, intOrder=2000, lastExchangeTime=1573779246139, loc=false, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false]]
Nov 15, 2019 @ 09:01:26.612 Nov 15, 2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger info
Nov 15, 2019 @ 09:01:26.612 INFO: New next node [newNext=TcpDiscoveryNode [id=6fcccf11-f903-4b4a-bbac-730ca0b80ce8, addrs=[10.254.169.217, 127.0.0.1], sockAddrs=[/127.0.0.1:47500, ignite-sit-4.ignite-sit.sit.svc.cluster.local/10.254.169.217:47500], discPort=47500, order=3912, intOrder=1993, lastExchangeTime=1573779190075, loc=false, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false]]
Nov 15, 2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Finished serving remote node connection [rmtAddr=/10.254.174.226:45453, rmtPort=45453
Nov 15, 2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger error

很抱歉日志格式错误。

我想知道导致 OutOfMemory 错误的原因以及如何防止这种情况再次发生。

非常感谢您的帮助。

更新:Heapdump分析结果:

The thread org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader @ 0xd9fbe2d0 tcp-disco-sock-reader-#369 keeps local variables with total size 312,295,344 (48.95%) bytes.

看起来 TCP SocketReader 需要大量堆内存。

Answer 1

调整 Java 堆设置、Pod 资源请求和 Pod 资源限制的某些组合。您可能需要设置所有三个，但具体细节取决于您的工作负载。