java - 如何解决 ThreadPoolExecutor 程序中的 java.lang.outOfMemoryError？

Question

我正在运行一个 ThreadPoolExecutor 程序，它创建 1000 个 url 并发送到执行程序服务。

Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:673)
    at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
    at java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650)
    at MyProgramName.main(MyProgramName.java:175)
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
2014-08-16 20:46:21.341 java[5100:d07] java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:673)
    at sun.awt.AWTAutoShutdown.activateBlockerThread(AWTAutoShutdown.java:312)
    at sun.awt.AWTAutoShutdown.setToolkitBusy(AWTAutoShutdown.java:232)
    at sun.awt.AWTAutoShutdown.notifyToolkitThreadBusy(AWTAutoShutdown.java:118)
2014-08-16 20:46:21.342 java[5100:d07] (
    0   CoreFoundation                      0x00007fff9127825c __exceptionPreprocess + 172
    1   libobjc.A.dylib                     0x00007fff9031ce75 objc_exception_throw + 43
    2   CoreFoundation                      0x00007fff91277e09 -[NSException raise] + 9
    3   JavaNativeFoundation                0x000000010c754c3f JNFCallStaticVoidMethod + 213
    4   libawt.jnilib                       0x0000000115fb8151 setBusy + 53
    5   CoreFoundation                      0x00007fff911a8d67 __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    6   CoreFoundation                      0x00007fff911a8cd7 __CFRunLoopDoObservers + 391
    7   CoreFoundation                      0x00007fff9119a608 __CFRunLoopRun + 1368
    8   CoreFoundation                      0x00007fff91199e75 CFRunLoopRunSpecific + 309
    9   HIToolbox                           0x00007fff8fdd9a0d RunCurrentEventLoopInMode + 226
    10  HIToolbox                           0x00007fff8fdd97b7 ReceiveNextEventCommon + 479
    11  HIToolbox                           0x00007fff8fdd95bc _BlockUntilNextEventMatchingListInModeWithFilter + 65
    12  AppKit                              0x00007fff8ac5d24e _DPSNextEvent + 1434
    13  AppKit                              0x00007fff8ac5c89b -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 122
    14  libawt.jnilib                       0x0000000115fb987f -[NSApplicationAWT nextEventMatchingMask:untilDate:inMode:dequeue:] + 124
    15  AppKit                              0x00007fff8ac5099c -[NSApplication run] + 553
    16  libawt.jnilib                       0x0000000115fb7f30 +[AWTStarter startAWT:] + 1495
    17  libawt.jnilib                       0x0000000115fb78aa -[CPerformer perform] + 93
    18  Foundation                          0x00007fff8c64313e __NSThreadPerformPerform + 229
    19  CoreFoundation                      0x00007fff911a95b1 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
    20  CoreFoundation                      0x00007fff9119ac62 __CFRunLoopDoSources0 + 242
    21  CoreFoundation                      0x00007fff9119a3ef __CFRunLoopRun + 831
    22  CoreFoundation                      0x00007fff91199e75 CFRunLoopRunSpecific + 309
    23  java                                0x000000010be86843 java + 18499
    24  java                                0x000000010be8629a java + 17050
    25  java                                0x000000010be83a98 java + 6808
    26  ???                                 0x0000000000000005 0x0 + 5
)
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL

我已经对 Internet 和 stackoverflow 进行了一些研究。一些答案说我需要减少堆栈大小。有人说我需要使用 ulimit -a 增加 ulimit。我完全糊涂了。我对这些答案感到困惑，因为它们并非特定于 ThreadExecutor。请帮助我

我在 Mac OS、i7 四核处理器、16 gb RAM、1 Tb HD 中运行代码

我的程序代码在这个问题中Why does the following executor service java Thread program doesn't shut down?

Answer 1

真正的问题是您试图使用过多的堆栈内存创建过多的线程。

您可以减少线程堆栈的默认大小(如建议的那样)，但这不是一个完整的解决方案。 (当你决定在更多的 URL 上运行你的程序时，你会再次遇到同样的问题。最终你会撞到“墙”，你不能进一步减少堆栈大小；例如，因为你正在获取 StackOverflowError 异常。)

更好的解决方案是不要创建那么多线程。您已将程序配置为使用无限 线程池。坦率地说，这很荒谬。改变这个:

ThreadPoolExecutor executor = 
      new ThreadPoolExecutor(Integer.MAX_VALUE, Integer.MAX_VALUE, 
                             20, TimeUnit.MILLISECONDS, blockingQueue);

更现实的东西:

ThreadPoolExecutor executor = 
      new ThreadPoolExecutor(100, 100, 
                             20, TimeUnit.MILLISECONDS, blockingQueue);

超过某个点，添加额外的线程不会使应用程序运行得更快。事实上，它通常会开始使应用程序运行变慢，有时灾难性地变慢。