java - 没有堆转储的 oom 错误 ec2 导致 jvm 崩溃

Question

服务器崩溃时出现以下错误

我发现的一件重要事情是分配失败值在每次垃圾收集后都会增加，并且在某些时候 jvm 会崩溃。对此有任何想法

101402.988: [GC (Allocation Failure) 101402.988: [DefNew: 207886K->12434K(219456K), 0.0330469 secs]101403.021: [Tenured: 488730K->418985K(488808K), 0.8195576 secs] 694094K->418985K(708264K), [Metaspace: 247627K->247627K(1263616K)]OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000f32aa000, 214532096, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 214532096 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/local/tomcat/logs/hs_err_pid16209.log

hs 错误文件总是相同的错误我们已经尝试增加 ram 并探索解决此问题的各种选项但问题还是一样

        There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (mmap) failed to map 214532096 bytes for committing reserved memory.
    # Possible reasons:


    #   The system is out of physical RAM or swap space
    #   In 32 bit mode, the process size limit was hit
    # Possible solutions:
    #   Reduce memory load on the system
    #   Increase physical memory or swap space
    #   Check if swap backing store is full
    #   Use 64 bit Java on a 64 bit OS
    #   Decrease Java heap size (-Xmx/-Xms)
    #   Decrease number of Java threads
    #   Decrease Java thread stack sizes (-Xss)
    #   Set larger code cache with -XX:ReservedCodeCacheSize=
    # This output file may be truncated or incomplete.
    #
    #  Out of Memory Error (os_linux.cpp:2638), pid=16209, tid=140213871417088
    #
    # JRE version: OpenJDK Runtime Environment (8.0_71-b15) (build 1.8.0_71-b15)
    # Java VM: OpenJDK 64-Bit Server VM (25.71-b15 mixed mode linux-amd64 compressed oops)
    # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
    #

    ---------------  T H R E A D  ---------------

    Current thread (0x00007f86100a4000):  VMThread [stack: 0x00007f8615ef9000,0x00007f8615ffa000] [id=16211]

    Stack: [0x00007f8615ef9000,0x00007f8615ffa000],  sp=0x00007f8615ff8510,  free space=1021k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    V  [libjvm.so+0xa0854d]
    V  [libjvm.so+0x4c4cca]
    V  [libjvm.so+0x86c2d0]
    V  [libjvm.so+0x8669ce]
    V  [libjvm.so+0xa05344]
    V  [libjvm.so+0x5af7ed]
    V  [libjvm.so+0x5afb6d]
    V  [libjvm.so+0x5a01e0]
    V  [libjvm.so+0x4570a2]
    V  [libjvm.so+0xa094f1]
    V  [libjvm.so+0xa0e707]
    V  [libjvm.so+0xa0d0f8]
    V  [libjvm.so+0xa0d60a]
    V  [libjvm.so+0xa0da21]
    V  [libjvm.so+0x869702]

    VM_Operation (0x00007f85ed7ace50): GenCollectForAllocation, mode: safepoint, requested by thread 0x00007f85d14e4000


    ---------------  P R O C E S S  ---------------

    Java Threads: ( => current thread )
      0x00007f85d04b5800 JavaThread "http-bio-8080-exec-699" daemon [_thread_blocked, id=30034, stack(0x00007f85cb0eb000,0x00007f85cb1ec000)]
      0x00007f85daaa8800 JavaThread "http-bio-8080-exec-700" daemon [_thread_blocked, id=30033, stack(0x00007f85cbee1000,0x00007f85cbfe2000)]
      0x00007f85d19e4000 JavaThread "http-bio-8080-exec-701" daemon [_thread_blocked, id=30032, stack(0x00007f85cc0e3000,0x00007f85cc1e4000)]
      0x00007f85d1ab3800 JavaThread "http-bio-8080-exec-702" daemon [_thread_blocked, id=30031, stack(0x00007f85cc7ea000,0x00007f85cc8eb000)]
      0x00007f85db4c5000 JavaThread "http-bio-8080-exec-714" daemon [_thread_blocked, id=30030, stack(0x00007f85cc8eb000,0x00007f85cc9ec000)]
      0x00007f85d19e5800 JavaThread "http-bio-8080-exec-713" daemon [_thread_blocked, id=30029, stack(0x00007f85ccddc000,0x00007f85ccedd000)]
      0x00007f85d14e4000 JavaThread "http-bio-8080-exec-712" daemon [_thread_blocked, id=30028, stack(0x00007f85ed6af000,0x00007f85ed7b0000)]
      0x00007f85db640800 JavaThread "http-bio-8080-exec-711" daemon [_thread_blocked, id=30027, stack(0x00007f85ed7b0000,0x00007f85ed8b1000)]
      0x00007f85dade1000 JavaThread "http-bio-8080-exec-710" daemon [_thread_blocked, id=30026, stack(0x00007f85ed8b1000,0x00007f85ed9b2000)]
      0x00007f85d8de8000 JavaThread "http-bio-8080-exec-709" daemon [_thread_blocked, id=30025, stack(0x00007f86000e8000,0x00007f86001e9000)]
      0x00007f85d0a29800 JavaThread "http-bio-8080-exec-708" daemon [_thread_blocked, id=30024, stack(0x00007f85c74b9000,0x00007f85c75ba000)]
      0x00007f85da827000 JavaThread "http-bio-8080-exec-707" daemon [_thread_blocked, id=30023, stack(0x00007f85c7dc2000,0x00007f85c7ec3000)]
      0x00007f85d946b000 JavaThread "http-bio-8080-exec-706" daemon [_thread_blocked, id=30022, stack(0x00007f85c96d1000,0x00007f85c97d2000)]
      0x00007f85d17a6800 JavaThread "http-bio-8080-exec-705" daemon [_thread_blocked, id=30021, stack(0x00007f85c5297000,0x00007f85c5398000)]
      0x00007f85d9a83000 JavaThread "http-bio-8080-exec-704" daemon [_thread_blocked, id=30020, stack(0x00007f85c6aaf000,0x00007f85c6bb0000)]
      0x00007f85d18ff800 JavaThread "http-bio-8080-exec-703" daemon [_thread_blocked, id=30019, stack(0x00007f85c8dc8000,0x00007f85c8ec9000)]
      0x00007f85d293a000 JavaThread "http-bio-8080-exec-698" daemon [_thread_blocked, id=30018, stack(0x00007f85ca7e2000,0x00007f85ca8e3000)]
      0x00007f85da19b800 JavaThread "http-bio-8080-exec-697" daemon [_thread_blocked, id=30007, stack(0x00007f85c76bb000,0x00007f85c77bc000)]
      0x00007f85dbd89000 JavaThread "http-bio-8080-exec-696" daemon [_thread_blocked, id=30005, stack(0x00007f85c70b5000,0x00007f85c71b6000)]
      0x00007f85daea1800 JavaThread "http-bio-8080-exec-695" daemon [_thread_blocked, id=30003, stack(0x00007f85c3e83000,0x00007f85c3f84000)]
      0x00007f85d07a3000 JavaThread "http-bio-8080-exec-694" daemon [_thread_blocked, id=30002, stack(0x00007f85cb4ed000,0x00007f85cb5ee000)]
      0x00007f85dbaa7800 JavaThread "http-bio-8080-exec-693" daemon [_thread_blocked, id=30001, stack(0x00007f85c357a000,0x00007f85c367b000)]
      0x00007f85dbe7c000 JavaThread "http-bio-8080-exec-692" daemon [_thread_blocked, id=30000, stack(0x00007f85cd0f3000,0x00007f85cd1f4000)]
      0x00007f85dbbd3000 JavaThread "http-bio-8080-exec-691" daemon [_thread_blocked, id=29999, stack(0x00007f85c3b80000,0x00007f85c3c81000)]
      0x00007f85d82ab800 JavaThread "http-bio-8080-exec-690" daemon [_thread_blocked, id=29998, stack(0x00007f85c8ec9000,0x00007f85c8fca000)]
      0x00007f85d06b3000 JavaThread "http-bio-8080-exec-689" daemon [_thread_blocked, id=29997, stack(0x00007f85c3a7f000,0x00007f85c3b80000)]
      0x00007f85db4c9000 JavaThread "http-bio-8080-exec-688" daemon [_thread_blocked, id=29996, stack(0x00007f85c9edb000,0x00007f85c9fdc000)]
      0x00007f85db8fd000 JavaThread "http-bio-8080-exec-687" daemon [_thread_blocked, id=29995, stack(0x00007f85c73b8000,0x00007f85c74b9000)]
      0x00007f85d86fa800 JavaThread "http-bio-8080-exec-686" daemon [_thread_blocked, id=29994, stack(0x00007f85c3479000,0x00007f85c357a000)]
      0x00007f85d071f000 JavaThread "http-bio-8080-exec-685" daemon [_thread_blocked, id=29993, stack(0x00007f85c72b7000,0x00007f85c73b8000)]
      0x00007f85d0fb2000 JavaThread "http-bio-8080-exec-684" daemon [_thread_blocked, id=29992, stack(0x00007f85cc6e7000,0x00007f85cc7e8000)]
      0x00007f85da5b4800 JavaThread "http-bio-8080-exec-683" daemon [_thread_blocked, id=29991, stack(0x00007f85c6bb0000,0x00007f85c6cb1000)]
      0x00007f85da8cb000 JavaThread "http-bio-8080-exec-682" daemon [_thread_blocked, id=29990, stack(0x00007f85c5ca1000,0x00007f85c5da2000)]
      0x00007f85d8f6e000 JavaThread "http-bio-8080-exec-677" daemon [_thread_blocked, id=29988, stack(0x00007f85c77bc000,0x00007f85c78bd000)]
      0x00007f85d19fe000 JavaThread "http-bio-8080-exec-681" daemon [_thread_blocked, id=29987, stack(0x00007f85c93ce000,0x00007f85c94cf000)]
      0x00007f85d0fa5000 JavaThread "http-bio-8080-exec-680" daemon [_thread_blocked, id=29986, stack(0x00007f85c387d000,0x00007f85c397e000)]
      0x00007f85d019f000 JavaThread "http-bio-8080-exec-679" daemon [_thread_blocked, id=29985, stack(0x00007f85c3d82000,0x00007f85c3e83000)]
      0x00007f85d8b24000 JavaThread "http-bio-8080-exec-678" daemon [_thread_blocked, id=29984, stack(0x00007f85c4a8f000,0x00007f85c4b90000)]
      0x00007f85d1e6a800 JavaThread "http-bio-8080-exec-676" daemon [_thread_blocked, id=29983, stack(0x00007f85cbde0000,0x00007f85cbee1000)]
      0x00007f85d9f81000 JavaThread "http-bio-8080-exec-675" daemon [_thread_blocked, id=29981, stack(0x00007f85c92cd000,0x00007f85c93ce000)]
      0x00007f85daddf800 JavaThread "http-bio-8080-exec-674" daemon [_thread_blocked, id=29978, stack(0x00007f85c69ae000,0x00007f85c6aaf000)]
      0x00007f85d9abd000 JavaThread "http-bio-8080-exec-670" daemon [_thread_blocked, id=29977, stack(0x00007f85c599e000,0x00007f85c5a9f000)]
      0x00007f85d9a08000 JavaThread "http-bio-8080-exec-671" daemon [_thread_blocked, id=29976, stack(0x00007f85c5a9f000,0x00007f85c5ba0000)]
      0x00007f85db357800 JavaThread "http-bio-8080-exec-672" daemon [_thread_blocked, id=29975, stack(0x00007f85c4b90000,0x00007f85c4c91000)]
      0x00007f85d911a800 JavaThread "http-bio-8080-exec-673" daemon [_thread_blocked, id=29974, stack(0x00007f85c80c5000,0x00007f85c81c6000)]
      0x00007f85d8802000 JavaThread "http-bio-8080-exec-669" daemon [_thread_blocked, id=29973, stack(0x00007f85c5196000,0x00007f85c5297000)]
      0x00007f85db4c9800 JavaThread "pool-36-thread-1" [_thread_blocked, id=29961, stack(0x00007f85c61a6000,0x00007f85c62a7000)]
      0x00007f85d8226000 JavaThread "http-bio-8080-exec-662" daemon [_thread_blocked, id=29954, stack(0x00007f85c78bd000,0x00007f85c79be000)]
      0x00007f85dadcf000 JavaThread "http-bio-8080-exec-663" daemon [_thread_blocked, id=29953, stack(0x00007f85c397e000,0x00007f85c3a7f000)]
      0x00007f85d9d7d800 JavaThread "http-bio-8080-exec-664" daemon [_thread_blocked, id=29952, stack(0x00007f85ce6ed000,0x00007f85ce7ee000)]
      0x00007f85dbe7a800 JavaThread "http-bio-8080-exec-665" daemon [_thread_blocked, id=29951, stack(0x00007f85c5da2000,0x00007f85c5ea3000)]
      0x00007f85d8635000 JavaThread "http-bio-8080-exec-667" daemon [_thread_blocked, id=29949, stack(0x00007f85c3f84000,0x00007f85c4085000)]
      0x00007f85daa7b800 JavaThread "http-bio-8080-exec-661" daemon [_thread_blocked, id=29948, stack(0x00007f85c90cb000,0x00007f85c91cc000)]
      0x00007f85daa44800 JavaThread "http-bio-8080-exec-644" daemon [_thread_blocked, id=29910, stack(0x00007f85cbfe2000,0x00007f85cc0e3000)]
      0x00007f85db40c800 JavaThread "http-bio-8080-exec-649" daemon [_thread_blocked, id=29895, stack(0x00007f85c367b000,0x00007f85c377c000)]
      0x00007f85db4d4800 JavaThread "http-bio-8080-exec-648" daemon [_thread_blocked, id=29894, stack(0x00007f85cd6f7000,0x00007f85cd7f8000)]
      0x00007f85d04f4800 JavaThread "http-bio-8080-exec-646" daemon [_thread_blocked, id=29893, stack(0x00007f85cc1e4000,0x00007f85cc2e5000)]
      0x00007f85d90cb000 JavaThread "http-bio-8080-exec-642" daemon [_thread_blocked, id=29890, stack(0x00007f85cb3ec000,0x00007f85cb4ed000)]
      0x00007f85db40b000 JavaThread "http-bio-8080-exec-637" daemon [_thread_blocked, id=29885, stack(0x00007f85c71b6000,0x00007f85c72b7000)]



    7f861a0e3000-7f861a0e4000 r--s 00001000 ca:01 310384983                  /usr/local/tomcat/lib/tomcat-api.jar
7f861a0e4000-7f861a0e6000 r--s 00001000 ca:01 327158654                  /usr/local/tomcat/temp/newrelic-bootstrap5567724053540453581.jar
7f861a0e6000-7f861a0ea000 r--s 000b0000 ca:01 210043060                  /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/jsse.jar
7f861a0ea000-7f861a1bb000 rw-p 00000000 00:00 0 
7f861a1bb000-7f861a1c3000 rw-s 00000000 ca:01 402669995                  /tmp/hsperfdata_tomcat/16209
7f861a1c3000-7f861a1c6000 ---p 00000000 00:00 0 
7f861a1c6000-7f861a2c9000 rw-p 00000000 00:00 0                          [stack:16210]
7f861a2c9000-7f861a2ca000 r--s 00000000 ca:01 327158655                  /usr/local/tomcat/temp/instrumentation4280289022478804408.jar
7f861a2ca000-7f861a2cb000 r--s 00003000 ca:01 310384972                  /usr/local/tomcat/lib/annotations-api.jar
7f861a2cb000-7f861a2cd000 r--s 00008000 ca:01 293631274                  /usr/local/tomcat/bin/tomcat-juli.jar
7f861a2cd000-7f861a2ce000 r--s 00005000 ca:01 293631268                  /usr/local/tomcat/bin/commons-daemon.jar
7f861a2ce000-7f861a2cf000 r--s 00006000 ca:01 293631264                  /usr/local/tomcat/bin/bootstrap.jar
7f861a2cf000-7f861a2d0000 rw-p 00000000 00:00 0 
7f861a2d0000-7f861a2d1000 ---p 00000000 00:00 0 
7f861a2d1000-7f861a2d2000 rw-p 00000000 00:00 0 
7f861a2d2000-7f861a2d3000 r--p 00021000 ca:01 1303                       /usr/lib64/ld-2.17.so
7f861a2d3000-7f861a2d4000 rw-p 00022000 ca:01 1303                       /usr/lib64/ld-2.17.so
7f861a2d4000-7f861a2d5000 rw-p 00000000 00:00 0 
7ffe366cd000-7ffe366ee000 rw-p 00000000 00:00 0                          [stack]
7ffe3670c000-7ffe3670e000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

VM Arguments:
jvm_args: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx1024m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -DOVERRIDE_COMPILED_CACHE=true -DclientUrl=https://katerra-dev.halosys.com/client-api -Dcom.halomem.redis=false -DDB_SERVER=localhost -DDB_PORT=5432 -DDB_NAME=halomem -DDB_USER=halosys -DDB_PASSWORD=halosys -DDB_MAX_POOL=30 -DCONFIG_FILE=classpath:/hm.properties -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/dump/ -javaagent:/usr/local/tomcat/newrelic/newrelic.jar -Djava.endorsed.dirs=/usr/local/tomcat/endorsed -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp 
java_command: org.apache.catalina.startup.Bootstrap start
java_class_path (initial): /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar:/usr/local/tomcat/newrelic/newrelic.jar
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
SHELL=/bin/bash

Signal Handlers:
SIGSEGV: [libjvm.so+0xa09060], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xa09060], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x867b90], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x867b90], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x867b90], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x867b90], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x867a40], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [libjvm.so+0x867c50], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGINT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: [libjvm.so+0x867c50], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.so+0x867c50], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO


---------------  S Y S T E M  ---------------

OS:CentOS Linux release 7.1.1503 (Core) 

uname:Linux 3.10.0-229.20.1.el7.x86_64 #1 SMP Tue Nov 3 19:10:07 UTC 2015 x86_64
libc:glibc 2.17 NPTL 2.17 
rlimit: STACK 8192k, CORE 0k, NPROC 4096, NOFILE 26384, AS infinity
load average:0.40 0.10 0.07

/proc/meminfo:
MemTotal:        1883272 kB
MemFree:           81416 kB
MemAvailable:      68368 kB
Buffers:               0 kB
Cached:           249212 kB
SwapCached:            0 kB
Active:          1592104 kB
Inactive:         101724 kB
Active(anon):    1559520 kB
Inactive(anon):    68704 kB
Active(file):      32584 kB
Inactive(file):    33020 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               108 kB
Writeback:             0 kB
AnonPages:       1444672 kB
Mapped:           130564 kB
Shmem:            183536 kB
Slab:              51212 kB
SReclaimable:      20916 kB
SUnreclaim:        30296 kB
KernelStack:        7328 kB
PageTables:        16912 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      941636 kB
Committed_AS:    3306632 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        5220 kB
VmallocChunk:   34359732223 kB
HardwareCorrupted:     0 kB
AnonHugePages:    628736 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       40960 kB
DirectMap2M:     2056192 kB


CPU:total 1 (1 cores per cpu, 1 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, tsc, bmi1, bmi2

/proc/cpuinfo:
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
stepping    : 2
microcode   : 0x36
cpu MHz     : 2400.042
cache size  : 30720 KB
physical id : 0
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm xsaveopt fsgsbase bmi1 avx2 smep bmi2 erms invpcid
bogomips    : 4800.08
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 1883272k(81416k free), swap 0k(0k free)

vm_info: OpenJDK 64-Bit Server VM (25.71-b15) for linux-amd64 JRE (1.8.0_71-b15), built on Jan 21 2016 16:00:19 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-4)

time: Tue Apr 11 10:44:59 2017
elapsed time: 101403 seconds (1d 4h 10m 3s)

Answer 1

不会为内存不足相关错误生成堆转储。您需要使用 -XX:+HeapDumpOnOutOfMemoryError 来获取堆转储，前提是您已通过“ulimit -c unlimited”选项将限制设置为无限制