java - 虚拟内存每秒增加 - 内存不足错误

Question

我的 Linux 系统具有 8GB RAM(7.78 GB 实际内存和 1024 MB 交换内存)和 4 个内核。我正在使用以下命令运行 spring boot 应用程序

screen -dmS API-Server java -jar -verbose:gc
-Dspring.profiles.active=prods1 -Xms1024m -Xmx4096m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ssvm/heapdump -XX:OnOutOfMemoryError='jmap -histo -F %p' /home/ssvm/api/nt-api-v2-0.0.1-SNAPSHOT.jar

我在此服务器上有一个 Apache，所有 API 请求都将由 Apache 接收，Apache 将重定向到 Java API 服务器。2000 名用户将在高峰时段使用此 API。最大 RAM 使用量为 1.5 到 2.5 GB。但是虚拟内存每秒都在增加(在 TOP 上对其进行监控)。有一次，新的 api 请求出现内存不足错误。CPU 使用率在峰值时间为 50%，RAM 使用率为:Apache - 1.36 GB 和 Java - 1.60 GB

我使用 jmap 进行了堆转储，我使用 VisualVM 对其进行了分析，RAM 使用量为 1443 MB。我有免费 RAM 和 CPU 使用率。我没有看到代码级别的任何问题。我看到的唯一区别是，虚拟内存每秒都在增加，达到 10 GB、20 GB、26 GB。

对这个问题有什么想法吗？1)为什么虚拟内存不断增加？2) 内存不足错误的原因可能是什么？

root@ssvms2ubuntu:~# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31861
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 350000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31861
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


root@ssvms2ubuntu:~# java -XX:+PrintFlagsFinal -version | grep ThreadStackSize
     intx CompilerThreadStackSize                   = 0                                   {pd product}
     intx ThreadStackSize                           = 1024                                {pd product}
     intx VMThreadStackSize                         = 1024                                {pd product}
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)


Error log is generated as below (Last line of the log clearly shows, It has sufficient memory):

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (gcTaskThread.cpp:48), pid=14493, tid=0x00007fefe3a1e700
#
# JRE version:  (8.0_171-b11) (build )
# Java VM: OpenJDK 64-Bit Server VM (25.171-b11 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

---------------  T H R E A D  ---------------

Current thread (0x0000000000c07800):  JavaThread "Unknown thread" [_thread_in_vm, id=14493, stack(0x00007ffc0363f000,0x00007ffc0373f000)]

Stack: [0x00007ffc0363f000,0x00007ffc0373f000],  sp=0x00007ffc03739620,  free space=1001k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xa6db9b]
V  [libjvm.so+0x4c694c]
V  [libjvm.so+0x5a87ff]
V  [libjvm.so+0x5a7967]
V  [libjvm.so+0x8e0169]
V  [libjvm.so+0xa310f9]
V  [libjvm.so+0xa31342]
V  [libjvm.so+0x61133f]
V  [libjvm.so+0xa166ea]
V  [libjvm.so+0x68b8c2]  JNI_CreateJavaVM+0x62
C  [libjli.so+0x2f2e]
C  [libjli.so+0x84f6]
C  [libjli.so+0x4a44]
C  [libjli.so+0x55a0]  JLI_Launch+0x9d0
C  [java+0x621]
C  [libc.so.6+0x20830]  __libc_start_main+0xf0


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

Other Threads:

=>0x0000000000c07800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=14493, stack(0x00007ffc0363f000,0x00007ffc0373f000)]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Classes redefined (0 events):
No events

Internal exceptions (0 events):
No events

Events (0 events):
No events


Dynamic libraries:
00400000-00401000 r-xp 00000000 08:00 256826                             /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
00600000-00601000 r--p 00000000 08:00 256826                             /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
00601000-00602000 rw-p 00001000 08:00 256826                             /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
00be8000-00c28000 rw-p 00000000 00:00 0                                  [heap]
83600000-88a00000 rw-p 00000000 00:00 0 
88a00000-d6780000 ---p 00000000 00:00 0 
d6780000-d9180000 rw-p 00000000 00:00 0 
d9180000-100000000 ---p 00000000 00:00 0 
7fefd093d000-7fefd0967000 rw-p 00000000 00:00 0 
7fefd0967000-7fefd0bd6000 ---p 00000000 00:00 0 
7fefd0bd6000-7fefd0c00000 rw-p 00000000 00:00 0 
7fefd0c00000-7fefd0e6e000 ---p 00000000 00:00 0 
7fefd0e6e000-7fefd0e84000 rw-p 00000000 00:00 0 
7fefd0e84000-7fefd0fbb000 ---p 00000000 00:00 0 
7fefd0fbb000-7fefd0fc6000 rw-p 00000000 00:00 0 
7fefd0fc6000-7fefd137c000 ---p 00000000 00:00 0 
7fefd137c000-7fefd15ec000 rwxp 00000000 00:00 0 
7fefd15ec000-7fefe037c000 ---p 00000000 00:00 0 
7fefe037c000-7fefe0384000 r-xp 00000000 08:00 256865                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libzip.so
7fefe0384000-7fefe0583000 ---p 00008000 08:00 256865                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libzip.so
7fefe0583000-7fefe0584000 r--p 00007000 08:00 256865                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libzip.so
7fefe0584000-7fefe0585000 rw-p 00008000 08:00 256865                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libzip.so
7fefe0585000-7fefe0590000 r-xp 00000000 08:00 4286                       /lib/x86_64-linux-gnu/libnss_files-2.23.so
7fefe0590000-7fefe078f000 ---p 0000b000 08:00 4286                       /lib/x86_64-linux-gnu/libnss_files-2.23.so
7fefe078f000-7fefe0790000 r--p 0000a000 08:00 4286                       /lib/x86_64-linux-gnu/libnss_files-2.23.so
7fefe0790000-7fefe0791000 rw-p 0000b000 08:00 4286                       /lib/x86_64-linux-gnu/libnss_files-2.23.so
7fefe0791000-7fefe0797000 rw-p 00000000 00:00 0 
7fefe0797000-7fefe07a2000 r-xp 00000000 08:00 4290                       /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7fefe07a2000-7fefe09a1000 ---p 0000b000 08:00 4290                       /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7fefe09a1000-7fefe09a2000 r--p 0000a000 08:00 4290                       /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7fefe09a2000-7fefe09a3000 rw-p 0000b000 08:00 4290                       /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7fefe09a3000-7fefe09b9000 r-xp 00000000 08:00 4264                       /lib/x86_64-linux-gnu/libnsl-2.23.so
7fefe09b9000-7fefe0bb8000 ---p 00016000 08:00 4264                       /lib/x86_64-linux-gnu/libnsl-2.23.so
7fefe0bb8000-7fefe0bb9000 r--p 00015000 08:00 4264                       /lib/x86_64-linux-gnu/libnsl-2.23.so
7fefe0bb9000-7fefe0bba000 rw-p 00016000 08:00 4264                       /lib/x86_64-linux-gnu/libnsl-2.23.so
7fefe0bba000-7fefe0bbc000 rw-p 00000000 00:00 0 
7fefe0bbc000-7fefe0bc4000 r-xp 00000000 08:00 6115                       /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7fefe0bc4000-7fefe0dc3000 ---p 00008000 08:00 6115                       /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7fefe0dc3000-7fefe0dc4000 r--p 00007000 08:00 6115                       /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7fefe0dc4000-7fefe0dc5000 rw-p 00008000 08:00 6115                       /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7fefe0dc5000-7fefe0df2000 r-xp 00000000 08:00 256856                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libjava.so
7fefe0df2000-7fefe0ff1000 ---p 0002d000 08:00 256856                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libjava.so
7fefe0ff1000-7fefe0ff2000 r--p 0002c000 08:00 256856                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libjava.so
7fefe0ff2000-7fefe0ff4000 rw-p 0002d000 08:00 256856                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libjava.so
7fefe0ff4000-7fefe1003000 r-xp 00000000 08:00 256842                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libverify.so
7fefe1003000-7fefe1202000 ---p 0000f000 08:00 256842                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libverify.so
7fefe1202000-7fefe1204000 r--p 0000e000 08:00 256842                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libverify.so
7fefe1204000-7fefe1205000 rw-p 00010000 08:00 256842                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/libverify.so
7fefe1205000-7fefe120c000 r-xp 00000000 08:00 4292                       /lib/x86_64-linux-gnu/librt-2.23.so
7fefe120c000-7fefe140b000 ---p 00007000 08:00 4292                       /lib/x86_64-linux-gnu/librt-2.23.so
7fefe140b000-7fefe140c000 r--p 00006000 08:00 4292                       /lib/x86_64-linux-gnu/librt-2.23.so
7fefe140c000-7fefe140d000 rw-p 00007000 08:00 4292                       /lib/x86_64-linux-gnu/librt-2.23.so
7fefe140d000-7fefe1423000 r-xp 00000000 08:00 2256                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7fefe1423000-7fefe1622000 ---p 00016000 08:00 2256                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7fefe1622000-7fefe1623000 rw-p 00015000 08:00 2256                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7fefe1623000-7fefe172b000 r-xp 00000000 08:00 6553                       /lib/x86_64-linux-gnu/libm-2.23.so
7fefe172b000-7fefe192a000 ---p 00108000 08:00 6553                       /lib/x86_64-linux-gnu/libm-2.23.so
7fefe192a000-7fefe192b000 r--p 00107000 08:00 6553                       /lib/x86_64-linux-gnu/libm-2.23.so
7fefe192b000-7fefe192c000 rw-p 00108000 08:00 6553                       /lib/x86_64-linux-gnu/libm-2.23.so
7fefe192c000-7fefe1a9e000 r-xp 00000000 08:00 24061                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7fefe1a9e000-7fefe1c9e000 ---p 00172000 08:00 24061                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7fefe1c9e000-7fefe1ca8000 r--p 00172000 08:00 24061                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7fefe1ca8000-7fefe1caa000 rw-p 0017c000 08:00 24061                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7fefe1caa000-7fefe1cae000 rw-p 00000000 00:00 0 
7fefe1cae000-7fefe2907000 r-xp 00000000 08:00 256869                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
7fefe2907000-7fefe2b07000 ---p 00c59000 08:00 256869                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
7fefe2b07000-7fefe2b9a000 r--p 00c59000 08:00 256869                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
7fefe2b9a000-7fefe2bc3000 rw-p 00cec000 08:00 256869                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
7fefe2bc3000-7fefe2bf3000 rw-p 00000000 00:00 0 
7fefe2bf3000-7fefe2c0b000 r-xp 00000000 08:00 4289                       /lib/x86_64-linux-gnu/libpthread-2.23.so
7fefe2c0b000-7fefe2e0a000 ---p 00018000 08:00 4289                       /lib/x86_64-linux-gnu/libpthread-2.23.so
7fefe2e0a000-7fefe2e0b000 r--p 00017000 08:00 4289                       /lib/x86_64-linux-gnu/libpthread-2.23.so
7fefe2e0b000-7fefe2e0c000 rw-p 00018000 08:00 4289                       /lib/x86_64-linux-gnu/libpthread-2.23.so
7fefe2e0c000-7fefe2e10000 rw-p 00000000 00:00 0 
7fefe2e10000-7fefe2e13000 r-xp 00000000 08:00 6114                       /lib/x86_64-linux-gnu/libdl-2.23.so
7fefe2e13000-7fefe3012000 ---p 00003000 08:00 6114                       /lib/x86_64-linux-gnu/libdl-2.23.so
7fefe3012000-7fefe3013000 r--p 00002000 08:00 6114                       /lib/x86_64-linux-gnu/libdl-2.23.so
7fefe3013000-7fefe3014000 rw-p 00003000 08:00 6114                       /lib/x86_64-linux-gnu/libdl-2.23.so
7fefe3014000-7fefe302d000 r-xp 00000000 08:00 2375                       /lib/x86_64-linux-gnu/libz.so.1.2.8
7fefe302d000-7fefe322c000 ---p 00019000 08:00 2375                       /lib/x86_64-linux-gnu/libz.so.1.2.8
7fefe322c000-7fefe322d000 r--p 00018000 08:00 2375                       /lib/x86_64-linux-gnu/libz.so.1.2.8
7fefe322d000-7fefe322e000 rw-p 00019000 08:00 2375                       /lib/x86_64-linux-gnu/libz.so.1.2.8
7fefe322e000-7fefe33ee000 r-xp 00000000 08:00 6558                       /lib/x86_64-linux-gnu/libc-2.23.so
7fefe33ee000-7fefe35ee000 ---p 001c0000 08:00 6558                       /lib/x86_64-linux-gnu/libc-2.23.so
7fefe35ee000-7fefe35f2000 r--p 001c0000 08:00 6558                       /lib/x86_64-linux-gnu/libc-2.23.so
7fefe35f2000-7fefe35f4000 rw-p 001c4000 08:00 6558                       /lib/x86_64-linux-gnu/libc-2.23.so
7fefe35f4000-7fefe35f8000 rw-p 00000000 00:00 0 
7fefe35f8000-7fefe3605000 r-xp 00000000 08:00 256866                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/jli/libjli.so
7fefe3605000-7fefe3805000 ---p 0000d000 08:00 256866                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/jli/libjli.so
7fefe3805000-7fefe3806000 r--p 0000d000 08:00 256866                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/jli/libjli.so
7fefe3806000-7fefe3807000 rw-p 0000e000 08:00 256866                     /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/jli/libjli.so
7fefe3807000-7fefe382d000 r-xp 00000000 08:00 4267                       /lib/x86_64-linux-gnu/ld-2.23.so
7fefe3914000-7fefe391c000 rw-s 00000000 08:00 375248                     /tmp/hsperfdata_ssvm/14493
7fefe391c000-7fefe391d000 ---p 00000000 00:00 0 
7fefe391d000-7fefe3a22000 rw-p 00000000 00:00 0 
7fefe3a28000-7fefe3a29000 rw-p 00000000 00:00 0 
7fefe3a29000-7fefe3a2a000 r--p 00000000 00:00 0 
7fefe3a2a000-7fefe3a2c000 rw-p 00000000 00:00 0 
7fefe3a2c000-7fefe3a2d000 r--p 00025000 08:00 4267                       /lib/x86_64-linux-gnu/ld-2.23.so
7fefe3a2d000-7fefe3a2e000 rw-p 00026000 08:00 4267                       /lib/x86_64-linux-gnu/ld-2.23.so
7fefe3a2e000-7fefe3a2f000 rw-p 00000000 00:00 0 
7ffc0363f000-7ffc03642000 ---p 00000000 00:00 0 
7ffc03642000-7ffc0373f000 rw-p 00000000 00:00 0                          [stack]
7ffc0377f000-7ffc03782000 r--p 00000000 00:00 0                          [vvar]
7ffc03782000-7ffc03784000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

VM Arguments:
java_command: <unknown>
java_class_path (initial): .
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=/home/ssvm/bin:/home/ssvm/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
SHELL=/bin/bash

Signal Handlers:
SIGSEGV: [libjvm.so+0xa6e6f0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xa6e6f0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x8b5d50], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x8b5d50], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x8b5d50], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x8b5d50], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x8b5c00], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none


---------------  S Y S T E M  ---------------

OS:DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"

uname:Linux 4.15.13-x86_64-linode106 #1 SMP Tue Mar 27 14:42:14 UTC 2018 x86_64
libc:glibc 2.23 NPTL 2.23 
rlimit: STACK 8192k, CORE 0k, NPROC 30000, NOFILE 350000, AS infinity
load average:0.12 0.61 0.78

/proc/meminfo:
MemTotal:        8162148 kB
MemFree:          926104 kB
MemAvailable:    4199816 kB
Buffers:           36152 kB
Cached:          3239184 kB
SwapCached:            0 kB
Active:          3805844 kB
Inactive:        2607852 kB
Active(anon):    3146828 kB
Inactive(anon):    22700 kB
Active(file):     659016 kB
Inactive(file):  2585152 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        262140 kB
SwapFree:         262140 kB
Dirty:               312 kB
Writeback:             0 kB
AnonPages:       3138376 kB
Mapped:            78500 kB
Shmem:             31172 kB
Slab:             258780 kB
SReclaimable:     136292 kB
SUnreclaim:       122488 kB
KernelStack:      207616 kB
PageTables:        69452 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4343212 kB
Committed_AS:   14450168 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
DirectMap4k:       40824 kB
DirectMap2M:     5201920 kB
DirectMap1G:     5242880 kB


CPU:total 4 (initial active 4) (1 cores per cpu, 1 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, tsc, bmi1, bmi2

/proc/cpuinfo:
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
stepping    : 2
microcode   : 0x1
cpu MHz     : 2499.988
cache size  : 16384 KB
physical id : 0
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs        : cpu_meltdown spectre_v1 spectre_v2
bogomips    : 5001.30
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
stepping    : 2
microcode   : 0x1
cpu MHz     : 2499.988
cache size  : 16384 KB
physical id : 1
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 1
initial apicid  : 1
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs        : cpu_meltdown spectre_v1 spectre_v2
bogomips    : 5001.30
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 2
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
stepping    : 2
microcode   : 0x1
cpu MHz     : 2499.988
cache size  : 16384 KB
physical id : 2
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 2
initial apicid  : 2
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs        : cpu_meltdown spectre_v1 spectre_v2
bogomips    : 5001.30
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 3
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
stepping    : 2
microcode   : 0x1
cpu MHz     : 2499.988
cache size  : 16384 KB
physical id : 3
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 3
initial apicid  : 3
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs        : cpu_meltdown spectre_v1 spectre_v2
bogomips    : 5001.30
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 8162148k(926104k free), swap 262140k(262140k free)

vm_info: OpenJDK 64-Bit Server VM (25.171-b11) for linux-amd64 JRE (1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11), built on Apr 27 2018 17:19:03 by "buildd" with gcc 5.4.0 20160609

time: Wed Jun 13 11:33:33 2018
elapsed time: 0 seconds (0d 0h 0m 0s)

Answer 1

应用程序正在使用创建线程并且在处理完成后不销毁线程的第三方库。这创造了巨大的活线程。所以虚拟内存增加，系统崩溃。

使用 jstack 找到它。

我们可以使用下面的命令来查看 linux 机器上正在运行的线程，ps- Sprite | wc -l

逐渐增加。

当我分析 jstack 报告时发现创建了巨大的线程并且没有正确销毁。