java - 如何调试为什么在 GC 期间停止线程花费了这么长时间？

Question

这个问题已经有答案了:

How do I get G1 to print more log details? (2 个回答)

已关闭 5 年前。

我正在运行 Cassandra 实例

$ java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b34)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b34, mixed mode)

以及以下垃圾收集选项

-Xms20000m
-Xmx20000m
-Xmn10000m
-Xss256k
-XX:+UseG1GC
-XX:MaxGCPauseMillis=1000
-XX:GCTimeRatio=97
-XX:MaxTenuringThreshold=2
-XX:+UseTLAB
-XX:+ResizeTLAB
-XX:+PerfDisableSharedMem
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintHeapAtGC
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintAdaptiveSizePolicy
-Xloggc:/dev/shm/cassandra-gc.log

在垃圾收集周期中，JVM 记录以下内容

2017-08-15T17:09:08.506+0000: 71712.670: [GC pause (G1 Evacuation Pause) (young)
Desired survivor size 658505728 bytes, new threshold 2 (max 2)
- age   1:   26818496 bytes,   26818496 total
- age   2:   14940744 bytes,   41759240 total
 71712.670: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 222556, predicted base time: 50.67 ms, remaining time: 949.33 ms, target pause time: 1000.00 ms]
 71712.670: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1242 regions, survivors: 8 regions, predicted young region time: 12.00 ms]
 71712.670: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1242 regions, survivors: 8 regions, old: 0 regions, predicted pause time: 62.68 ms, target pause time: 1000.00 ms]
 71712.736: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: recent GC overhead higher than threshold after GC, recent GC overhead: 1.39 %, threshold: 1.02 %, uncommitted: 0 bytes, calculated expansion amount: 0 bytes (20.00 %)]
, 0.0662411 secs]
   [Parallel Time: 57.5 ms, GC Workers: 33]
      [GC Worker Start (ms): Min: 71712670.9, Avg: 71712671.2, Max: 71712671.4, Diff: 0.4]
      [Ext Root Scanning (ms): Min: 6.4, Avg: 7.0, Max: 10.2, Diff: 3.8, Sum: 232.3]
      [Update RS (ms): Min: 34.0, Avg: 37.0, Max: 38.9, Diff: 4.9, Sum: 1221.2]
         [Processed Buffers: Min: 27, Avg: 48.5, Max: 66, Diff: 39, Sum: 1599]
      [Scan RS (ms): Min: 0.0, Avg: 0.2, Max: 0.5, Diff: 0.4, Sum: 5.1]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
      [Object Copy (ms): Min: 11.3, Avg: 12.6, Max: 12.9, Diff: 1.7, Sum: 415.0]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.4]
         [Termination Attempts: Min: 1, Avg: 2.1, Max: 4, Diff: 3, Sum: 68]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 4.5]
      [GC Worker Total (ms): Min: 56.7, Avg: 56.9, Max: 57.3, Diff: 0.7, Sum: 1878.5]
      [GC Worker End (ms): Min: 71712728.0, Avg: 71712728.1, Max: 71712728.3, Diff: 0.3]
   [Code Root Fixup: 0.5 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 1.1 ms]
   [Other: 7.1 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 2.7 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.8 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 1.5 ms]
   [Eden: 9936.0M(9936.0M)->0.0B(9920.0M) Survivors: 64.0M->80.0M Heap: 18.2G(19.5G)->8718.4M(19.5G)]
Heap after GC invocations=8876 (full 0):
 garbage-first heap   total 20480000K, used 8927594K [0x00000002de000000, 0x00000002de804e20, 0x00000007c0000000)
  region size 8192K, 10 young (81920K), 10 survivors (81920K)
 Metaspace       used 44399K, capacity 44978K, committed 45184K, reserved 1089536K
  class space    used 4591K, capacity 4729K, committed 4760K, reserved 1048576K
}
 [Times: user=0.88 sys=0.24, real=0.07 secs]
2017-08-15T17:09:08.573+0000: 71712.737: Total time for which application threads were stopped: 4.1542217 seconds, Stopping threads took: 4.0814458 seconds

我会提醒大家注意最后一行

Total time for which application threads were stopped: 4.1542217 seconds, Stopping threads took: 4.0814458 seconds

我无法从任何打印的 GC 详细信息中看出为什么停止线程需要这么长时间。

其他(较长的)暂停的线程停止时间非常短

2017-08-15T17:09:26.646+0000: 71730.810: Total time for which application threads were stopped: 0.1165019 seconds, Stopping threads took: 0.0005363 seconds
2017-08-15T17:10:36.982+0000: 71801.146: Total time for which application threads were stopped: 0.1444364 seconds, Stopping threads took: 0.0006902 seconds
2017-08-15T17:10:42.091+0000: 71806.255: Total time for which application threads were stopped: 0.2288663 seconds, Stopping threads took: 0.0064196 seconds
2017-08-15T17:10:52.601+0000: 71816.765: Total time for which application threads were stopped: 0.2781895 seconds, Stopping threads took: 0.0005429 seconds
2017-08-15T17:10:56.992+0000: 71821.156: Total time for which application threads were stopped: 0.1380046 seconds, Stopping threads took: 0.0006240 seconds

应用程序至少有 1000 个线程，但停止它们会占用 GC 事件持续时间的 98%，这对我来说毫无意义。

我首先假设这是线程到达安全点所需的时间，所以我尝试添加

-XX:+SafepointTimeout -XX:SafepointTimeoutDelay=500

但是记录的事件与长 GC 事件不一致(除非线程都需要 499 毫秒才能到达安全点...)

还有什么可以延迟线程停止时间？

请注意，无论我使用 CMS 还是 G1，也无论我 write the GC logs to disk (SSD) or to memory，这些暂停都会存在。 .

Answer 1

您可以添加 -XX:+PrintSafepointStatistics –XX:PrintSafepointStatisticsCount=1 以启用安全点的详细诊断。

-XX:+PrintJNIGCStalls 是另一个诊断 option这可能有助于解决问题。

JNIGCStalls 不计入 SafepointTimeout。

其他原因通常可能是系统过载。我建议您检查“平均负载”的峰值是否与异常 GC 相关。尽管“平均负载”指标有 quirks靠自己。

如果您处于虚拟化环境中，偶尔的系统卡住可能是一种常态。此类卡住也可以被视为“平均负载”操作系统指标的峰值。