java - 老一代总是 100% > 导致 100% CPU 使用率

Question

我的产品中运行着一个 Java 应用程序，它基本上是一个 Spring Rest MVC 项目。我观察到，我的应用程序的堆使用率从 10% 开始慢慢增加到最大 78%。

当达到78%时，我的节点的CPU使用率恒定为100%。看来 full GC 运行得太频繁了。

这是我的配置:

-XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSParallelRemarkEnabled -XX:+DoEscapeAnalysis -XX:InitialHeapSize=4294967296 -XX:LargePageSizeInBytes=268435456 -XX:+ManagementServer -XX:MaxHeapSize=12884901888 -XX:MaxNewSize=5368709120 -XX:MaxPermSize=1073741824 -XX:MaxTenuringThreshold=0 -XX:NewSize=536870912 -XX:OldPLABSize=16 -XX:ParallelGCThreads=12 -XX:PermSize=536870912 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:SurvivorRatio=16 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 

这是 Visual GC 的快照

您可以看到 CPU 使用率持续居高不下。以下是一些 GC 日志条目:

2017-10-02T08:54:42.768+0530: 163327.650: [CMS-concurrent-preclean-start]
2017-10-02T08:54:42.789+0530: 163327.671: [CMS-concurrent-preclean: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2017-10-02T08:54:42.789+0530: 163327.671: [CMS-concurrent-abortable-preclean-start]
2017-10-02T08:54:42.789+0530: 163327.671: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2017-10-02T08:54:42.790+0530: 163327.672: [GC[YG occupancy: 3025966 K (3466240 K)]2017-10-02T08:54:42.790+0530: 163327.672: [Rescan (parallel) , 6.7814290 secs]2017-10-02T08:54:49.571+0530: 163334.453: [weak refs processing, 0.0000540 secs]2017-10-02T08:54:49.572+0530: 163334.453: [scrub string table, 0.0013940 secs] [1 CMS-remark: 7340030K(7340032K)] 10365996K(10806272K), 6.7830680 secs] [Times: user=26.59 sys=0.48, real=6.79 secs]
2017-10-02T08:54:49.573+0530: 163334.455: [CMS-concurrent-sweep-start]
2017-10-02T08:54:53.196+0530: 163338.078: [CMS-concurrent-sweep: 3.618/3.623 secs] [Times: user=3.80 sys=0.02, real=3.62 secs]
2017-10-02T08:54:53.196+0530: 163338.078: [CMS-concurrent-reset-start]
2017-10-02T08:54:53.212+0530: 163338.094: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2017-10-02T08:54:54.143+0530: 163339.025: [GC [1 CMS-initial-mark: 7340030K(7340032K)] 10382251K(10806272K), 2.1178610 secs] [Times: user=2.12 sys=0.00, real=2.12 secs]
2017-10-02T08:54:56.261+0530: 163341.143: [CMS-concurrent-mark-start]
2017-10-02T08:55:00.966+0530: 163345.848: [CMS-concurrent-mark: 4.700/4.705 secs] [Times: user=14.18 sys=0.03, real=4.70 secs]
2017-10-02T08:55:00.966+0530: 163345.848: [CMS-concurrent-preclean-start]
2017-10-02T08:55:00.987+0530: 163345.869: [CMS-concurrent-preclean: 0.020/0.021 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2017-10-02T08:55:00.987+0530: 163345.869: [CMS-concurrent-abortable-preclean-start]
2017-10-02T08:55:00.987+0530: 163345.869: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2017-10-02T08:55:00.988+0530: 163345.870: [GC[YG occupancy: 3046326 K (3466240 K)]2017-10-02T08:55:00.988+0530: 163345.870: [Rescan (parallel) , 7.3009290 secs]2017-10-02T08:55:08.289+0530: 163353.171: [weak refs processing, 0.0000540 secs]2017-10-02T08:55:08.289+0530: 163353.171: [scrub string table, 0.0013900 secs] [1 CMS-remark: 7340030K(7340032K)] 10386356K(10806272K), 7.3026000 secs] [Times: user=28.62 sys=0.52, real=7.31 secs]
2017-10-02T08:55:08.291+0530: 163353.173: [CMS-concurrent-sweep-start]
2017-10-02T08:55:12.068+0530: 163356.950: [CMS-concurrent-sweep: 3.773/3.777 secs] [Times: user=3.96 sys=0.04, real=3.77 secs]
2017-10-02T08:55:12.068+0530: 163356.950: [CMS-concurrent-reset-start]
2017-10-02T08:55:12.084+0530: 163356.966: [CMS-concurrent-reset: 0.016/0.016 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2017-10-02T08:55:12.140+0530: 163357.021: [GC [1 CMS-initial-mark: 7340030K(7340032K)] 10397770K(10806272K), 2.1250470 secs] [Times: user=2.13 sys=0.00, real=2.13 secs]
2017-10-02T08:55:14.265+0530: 163359.147: [CMS-concurrent-mark-start]
2017-10-02T08:55:18.472+0530: 163363.354: [CMS-concurrent-mark: 4.202/4.207 secs] [Times: user=12.73 sys=0.03, real=4.21 secs]
2017-10-02T08:55:18.472+0530: 163363.354: [CMS-concurrent-preclean-start]
2017-10-02T08:55:18.498+0530: 163363.380: [CMS-concurrent-preclean: 0.025/0.026 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
2017-10-02T08:55:18.498+0530: 163363.380: [CMS-concurrent-abortable-preclean-start]
2017-10-02T08:55:18.498+0530: 163363.380: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2017-10-02T08:55:18.499+0530: 163363.381: [GC[YG occupancy: 3065834 K (3466240 K)]2017-10-02T08:55:18.499+0530: 163363.381: [Rescan (parallel) , 8.9283020 secs]2017-10-02T08:55:27.427+0530: 163372.309: [weak refs processing, 0.0000510 secs]2017-10-02T08:55:27.427+0530: 163372.309: [scrub string table, 0.0013920 secs] [1 CMS-remark: 7340030K(7340032K)] 10405864K(10806272K), 8.9299040 secs] [Times: user=34.99 sys=0.66, real=8.93 secs]
2017-10-02T08:55:27.429+0530: 163372.311: [CMS-concurrent-sweep-start]
2017-10-02T08:55:31.125+0530: 163376.007: [CMS-concurrent-sweep: 3.694/3.696 secs] [Times: user=3.80 sys=0.02, real=3.70 secs]
2017-10-02T08:55:31.126+0530: 163376.007: [CMS-concurrent-reset-start]
2017-10-02T08:55:31.141+0530: 163376.023: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2017-10-02T08:55:31.203+0530: 163376.085: [GC [1 CMS-initial-mark: 7340030K(7340032K)] 10409819K(10806272K), 2.1555220 secs] [Times: user=2.16 sys=0.00, real=2.15 secs]
2017-10-02T08:55:33.359+0530: 163378.241: [CMS-concurrent-mark-start]
2017-10-02T08:55:37.571+0530: 163382.453: [CMS-concurrent-mark: 4.205/4.212 secs] [Times: user=12.66 sys=0.02, real=4.22 secs]
2017-10-02T08:55:37.571+0530: 163382.453: [CMS-concurrent-preclean-start]
2017-10-02T08:55:37.591+0530: 163382.473: [CMS-concurrent-preclean: 0.020/0.020 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2017-10-02T08:55:37.591+0530: 163382.473: [CMS-concurrent-abortable-preclean-start]
2017-10-02T08:55:37.591+0530: 163382.473: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2017-10-02T08:55:37.592+0530: 163382.474: [GC[YG occupancy: 3071464 K (3466240 K)]2017-10-02T08:55:37.592+0530: 163382.474: [Rescan (parallel) , 7.4793360 secs]2017-10-02T08:55:45.071+0530: 163389.953: [weak refs processing, 0.0000660 secs]2017-10-02T08:55:45.071+0530: 163389.953: [scrub string table, 0.0014980 secs] [1 CMS-remark: 7340030K(7340032K)] 10411494K(10806272K), 7.4810960 secs] [Times: user=29.32 sys=0.55, real=7.48 secs]
2017-10-02T08:55:45.073+0530: 163389.955: [CMS-concurrent-sweep-start]
2017-10-02T08:55:48.759+0530: 163393.641: [CMS-concurrent-sweep: 3.685/3.686 secs] [Times: user=3.75 sys=0.00, real=3.68 secs]
2017-10-02T08:55:48.759+0530: 163393.641: [CMS-concurrent-reset-start]
2017-10-02T08:55:48.774+0530: 163393.656: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2017-10-02T08:55:50.394+0530: 163395.276: [GC [1 CMS-initial-mark: 7340030K(7340032K)] 10412726K(10806272K), 2.1372650 secs] [Times: user=2.14 sys=0.00, real=2.14 secs]
2017-10-02T08:55:52.532+0530: 163397.414: [CMS-concurrent-mark-start]
2017-10-02T08:55:57.099+0530: 163401.981: [CMS-concurrent-mark: 4.562/4.567 secs] [Times: user=13.79 sys=0.03, real=4.56 secs]
2017-10-02T08:55:57.100+0530: 163401.981: [CMS-concurrent-preclean-start]
2017-10-02T08:55:57.121+0530: 163402.003: [CMS-concurrent-preclean: 0.021/0.021 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
2017-10-02T08:55:57.121+0530: 163402.003: [CMS-concurrent-abortable-preclean-start]
2017-10-02T08:55:57.121+0530: 163402.003: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2017-10-02T08:55:57.122+0530: 163402.003: [GC[YG occupancy: 3079753 K (3466240 K)]2017-10-02T08:55:57.122+0530: 163402.003: [Rescan (parallel) , 6.3679960 secs]2017-10-02T08:56:03.490+0530: 163408.371: [weak refs processing, 0.0000530 secs]2017-10-02T08:56:03.490+0530: 163408.372: [scrub string table, 0.0013790 secs] [1 CMS-remark: 7340030K(7340032K)] 10419783K(10806272K), 6.3695720 secs] [Times: user=24.99 sys=0.44, real=6.37 secs]
2017-10-02T08:56:03.491+0530: 163408.373: [CMS-concurrent-sweep-start]
2017-10-02T08:56:07.238+0530: 163412.120: [CMS-concurrent-sweep: 3.743/3.746 secs] [Times: user=3.82 sys=0.01, real=3.74 secs]
2017-10-02T08:56:07.238+0530: 163412.120: [CMS-concurrent-reset-start]
2017-10-02T08:56:07.253+0530: 163412.135: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2017-10-02T08:56:07.305+0530: 163412.187: [GC [1 CMS-initial-mark: 7340030K(7340032K)] 10421003K(10806272K), 2.2521140 secs] [Times: user=2.25 sys=0.00, real=2.25 secs]
2017-10-02T08:56:09.557+0530: 163414.439: [CMS-concurrent-mark-start]
2017-10-02T08:56:13.791+0530: 163418.673: [CMS-concurrent-mark: 4.228/4.234 secs] [Times: user=12.87 sys=0.03, real=4.24 secs]
2017-10-02T08:56:13.791+0530: 163418.673: [CMS-concurrent-preclean-start]
2017-10-02T08:56:13.812+0530: 163418.694: [CMS-concurrent-preclean: 0.020/0.021 secs] [Times: user=0.02 sys=0.01, real=0.02 secs]
2017-10-02T08:56:13.812+0530: 163418.694: [CMS-concurrent-abortable-preclean-start]
2017-10-02T08:56:13.812+0530: 163418.694: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2017-10-02T08:56:13.813+0530: 163418.694: [GC[YG occupancy: 3093202 K (3466240 K)]2017-10-02T08:56:13.813+0530: 163418.695: [Rescan (parallel) , 8.0104730 secs]2017-10-02T08:56:21.823+0530: 163426.705: [weak refs processing, 0.0000520 secs]2017-10-02T08:56:21.823+0530: 163426.705: [scrub string table, 0.0014030 secs] [1 CMS-remark: 7340030K(7340032K)] 10433232K(10806272K), 8.0121360 secs] [Times: user=31.38 sys=0.59, real=8.01 secs]
2017-10-02T08:56:21.825+0530: 163426.707: [CMS-concurrent-sweep-start]
2017-10-02T08:56:25.577+0530: 163430.458: [CMS-concurrent-sweep: 3.746/3.752 secs] [Times: user=3.91 sys=0.02, real=3.75 secs]
2017-10-02T08:56:25.577+0530: 163430.459: [CMS-concurrent-reset-start]
2017-10-02T08:56:25.592+0530: 163430.474: [CMS-concurrent-reset: 0.016/0.016 secs] [Times: user=0.01 sys=0.01, real=0.02 secs]
2017-10-02T08:56:26.334+0530: 163431.216: [GC [1 CMS-initial-mark: 7340030K(7340032K)] 10440089K(10806272K), 2.1770880 secs] [Times: user=2.18 sys=0.00, real=2.18 secs]
2017-10-02T08:56:28.511+0530: 163433.393: [CMS-concurrent-mark-start]
2017-10-02T08:56:33.019+0530: 163437.900: [CMS-concurrent-mark: 4.502/4.507 secs] [Times: user=13.64 sys=0.01, real=4.50 secs]
2017-10-02T08:56:33.019+0530: 163437.901: [CMS-concurrent-preclean-start]
2017-10-02T08:56:33.043+0530: 163437.925: [CMS-concurrent-preclean: 0.023/0.024 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
2017-10-02T08:56:33.043+0530: 163437.925: [CMS-concurrent-abortable-preclean-start]
2017-10-02T08:56:33.043+0530: 163437.925: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2017-10-02T08:56:33.043+0530: 163437.925: [GC[YG occupancy: 3107362 K (3466240 K)]2017-10-02T08:56:33.043+0530: 163437.925: [Rescan (parallel) , 8.2780770 secs]2017-10-02T08:56:41.322+0530: 163446.203: [weak refs processing, 0.0000520 secs]2017-10-02T08:56:41.322+0530: 163446.204: [scrub string table, 0.0013870 secs] [1 CMS-remark: 7340030K(7340032K)] 10447392K(10806272K), 8.2796480 secs] [Times: user=32.44 sys=0.62, real=8.28 secs]
2017-10-02T08:56:41.323+0530: 163446.205: [CMS-concurrent-sweep-start]
2017-10-02T08:56:45.033+0530: 163449.915: [CMS-concurrent-sweep: 3.705/3.710 secs] [Times: user=3.79 sys=0.01, real=3.71 secs]
2017-10-02T08:56:45.034+0530: 163449.915: [CMS-concurrent-reset-start]
2017-10-02T08:56:45.049+0530: 163449.931: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2017-10-02T08:56:45.392+0530: 163450.274: [GC [1 CMS-initial-mark: 7340030K(7340032K)] 10449182K(10806272K), 2.2074740 secs] [Times: user=2.21 sys=0.00, real=2.20 secs]

完整的 GC 日志在这里: https://www.dropbox.com/s/keo4i0ss2m60jlk/gc.log.zip?dl=0

系统是 16 GB 机器。我以前在 8GB 机器上运行没有任何问题。是否表示内存泄漏。

或者只是我需要增加系统内存？

Answer 1

The system is 16 GB machine. I used to run on 8GB machine without any issues. Does it indicate memory leak.

这并不一定表明这一点。但这也不排除它。例如，您可能更改了代码，或者您现在可能遇到了更大的“问题”。

没有直接、明确的证据可以区分内存泄漏和“内存不足”

如果您怀疑内存泄漏，那么您应该使用内存分析器运行应用程序，并尝试确定内存是否按照您预期的方式使用。

<小时/>

Or is it just that I need to increase the system memory?

好吧，如果问题是内存泄漏，那么这可能无法解决问题。

<小时/>

第三个选项可能是重新设计或调整应用程序以使用更少的内存。例如，如果它在内存中缓存内容，那么减少缓存大小将减少占用的内存。或者，如果它生成大量垃圾，那么您也许可以重新实现一些东西来减少垃圾。