java - 评估 - OutOfMemoryError : unable to create new native thread

Question

虽然这是各种 Stackoverflow 用户报告的典型错误消息，但我的问题与如何评估所提出的解决方案是否解决了问题有关。

我阅读了与此错误相关的各种讨论和文章，大多数解决方案都深入到 Linux ulimits，我想我也是如此。

我的 ulimit 值为:

STACK 10240k, CORE 0k, NPROC 1024, NOFILE 4096;

我猜问题可能出在 NOPROC/NOFILE 太低(只有默认值)。

但是，我想知道是否有一种确切的方法可以确定根本原因，比如已超过 NOPROC 等，以及是否有一种方法可以准确评估当前正在使用的进程/文件句柄数；或者还有其他一些我应该关注的问题可以进行统计评估吗？

仅供引用，发生此问题时，未启用堆转储，并且在错误点没有线程转储数据。

感谢您对评估和修复此问题的投入。

这是简短的堆栈跟踪:

Caused by: java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:714)

这是系统值:

OS:Red Hat Enterprise Linux Server release 6.3 (Santiago)
uname:Linux 2.6.32-279.el6.x86_64 #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64
libc:glibc 2.12 NPTL 2.12
rlimit: STACK 10240k, CORE 0k, NPROC 1024, NOFILE 4096, AS infinity
load average:0.11 0.10 0.03
CPU:total 32 (8 cores per cpu, 2 threads per core) family 6 model 45 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, ht, tsc, tscinvbit, tscinv

/proc/meminfo:
MemTotal:       74206252 kB
MemFree:         2788244 kB
Buffers:         1042212 kB
Cached:         58454988 kB
SwapCached:         2860 kB
Active:         38242540 kB
Inactive:       29129604 kB

这是来自 JVM 崩溃报告的信息 - hs_err_pidxxxxx.log:

# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
...
#  Out of Memory Error (gcTaskThread.cpp:46), pid=20396, tid=140365307795200

# JRE version:  (7.0_80-b15) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

Current thread (0x00007fa95400a800):  JavaThread "Unknown thread" [_thread_in_vm, id=20458, stack(0x00007fa9583f5000,0x00007fa9584f6000)]
Stack: [0x00007fa9583f5000,0x00007fa9584f6000],  sp=0x00007fa9584f4540,  free space=1021k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x9a320a]  VMError::report_and_die()+0x2ea
V  [libjvm.so+0x498d3b]  report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
V  [libjvm.so+0x55943a]  GCTaskThread::GCTaskThread(GCTaskManager*, unsigned int, unsigned int)+0x11a
V  [libjvm.so+0x5589b8]  GCTaskManager::initialize()+0x2b8
V  [libjvm.so+0x843438]  ParallelScavengeHeap::initialize()+0x6f8
V  [libjvm.so+0x97509a]  Universe::initialize_heap()+0xca
V  [libjvm.so+0x976269]  universe_init()+0x79
V  [libjvm.so+0x5b2f25]  init_globals()+0x65
V  [libjvm.so+0x95db4d]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x1ed
V  [libjvm.so+0x63b2e4]  JNI_CreateJavaVM+0x74
C  [libjli.so+0x2f8e]  JavaMain+0x9e
Java Threads: ( => current thread )
Other Threads:
=>0x00007fa95400a800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=20458, stack(0x00007fa9583f5000,0x00007fa9584f6000)]
VM state:not at safepoint (not fully initialized)
VM Mutex/Monitor currently owned by a thread: None
GC Heap History (0 events):
No events
Deoptimization events (0 events):
No events
Internal exceptions (0 events):
No events
Events (0 events):
No events

Answer 1

I wanted to know if there is an exact way to identify the root cause say the NOPROC has been exceeded etc

JVM 与任何其他软件一样，最终必须通过系统调用与内核对话。要生成新线程，它必须使用 clone可以返回各种错误代码的系统调用(在手册页中有记录)。您可以使用 strace记录系统调用并查看它们的错误代码，这可以提供比 OOME 更细粒度的信息。