multithreading - 如果一个线程调用getStackTrace()，而在另一个线程中发生lambda定义(通过Unsafe.defineAnonymousClass)，则Java8会挂断

Question

我的Web应用程序运行在16核(带有HT的32-核)Dual-Xeon NUMA计算机上的Apache Tomcat/8.0.21，JVM 1.8.0_45-b15和Windows Server 2012上，可能会卡住，这很不幸。在这种情况下，标题中描述的 Action 在两个不同的线程中同时发生。

执行第一个 Action (getStackTrace())的线程正在尝试执行一些诊断，以检测系统的哪一部分正在减慢速度并在调用Thread.dumpThreads时卡住。
另一个线程正在执行某些操作，其中包括部分JVM的底层lambda定义。

特别是，我具有以下堆栈跟踪(通过jstack -F <pid>获得):

Attaching to process ID 6568, please wait... 
Debugger attached successfully. 
Server compiler detected. 
JVM version is 25.45-b02 

Deadlock Detection:

No deadlocks found. (... well, that's not the kind of deadlock you were searching for, dear JVM, but something bad is happening altogether :( )

Thread 155: (state = BLOCKED)
 - sun.misc.Unsafe.defineAnonymousClass(java.lang.Class, byte[], java.lang.Object[]) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.invoke.InvokerBytecodeGenerator.loadAndInitializeInvokerClass(byte[], java.lang.Object[]) @bci=8 (Compiled frame)
 - java.lang.invoke.InvokerBytecodeGenerator.loadMethod(byte[]) @bci=6 (Compiled frame)
 - java.lang.invoke.InvokerBytecodeGenerator.generateCustomizedCode(java.lang.invoke.LambdaForm, java.lang.invoke.MethodType) @bci=17 (Compiled frame)
 - java.lang.invoke.LambdaForm.compileToBytecode() @bci=65 (Compiled frame)
 - java.lang.invoke.DirectMethodHandle.makePreparedLambdaForm(java.lang.invoke.MethodType, int) @bci=638 (Interpreted frame)
 - java.lang.invoke.DirectMethodHandle.preparedLambdaForm(java.lang.invoke.MethodType, int) @bci=17 (Compiled frame)
 - java.lang.invoke.DirectMethodHandle.preparedLambdaForm(java.lang.invoke.MemberName) @bci=163 (Compiled frame)
 - java.lang.invoke.DirectMethodHandle.make(byte, java.lang.Class, java.lang.invoke.MemberName) @bci=94 (Compiled frame)
 - java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(byte, java.lang.Class, java.lang.invoke.MemberName, boolean, boolean, java.lang.Class) @bci=201 (Compiled frame)
 - java.lang.invoke.MethodHandles$Lookup.getDirectMethodNoSecurityManager(byte, java.lang.Class, java.lang.invoke.MemberName, java.lang.Class) @bci=8 (Compiled frame)
 - java.lang.invoke.MethodHandles$Lookup.getDirectMethodForConstant(byte, java.lang.Class, java.lang.invoke.MemberName) @bci=30 (Compiled frame)
 - java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(byte, java.lang.Class, java.lang.String, java.lang.Object) @bci=115 (Compiled frame)
 - java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(java.lang.Class, int, java.lang.Class, java.lang.String, java.lang.Object) @bci=38 (Compiled frame)
 - c.e.s.w.t.si.a.DDVP.lambda$1(com.vaadin.data.Container, com.vaadin.ui.HorizontalLayout, com.vaadin.ui.Label, com.vaadin.ui.Label, java.lang.String, java.lang.String, java.util.Map) @bci=48, line=104 (Interpreted frame)
 - c.e.s.w.t.si.a.DDVP$$Lambda$637.updateUIWith(java.lang.Object) @bci=32 (Interpreted frame)
 - c.e.s.w.d.DU$VoidUIUpdaterFromUIUpdater.updateUI() @bci=8, line=321 (Compiled frame)
 - c.e.s.w.d.DU$CompletionSignallingVoidUIUpdater.updateUI() @bci=4, line=125 (Compiled frame)
 - c.e.s.w.d.CUQ$1.sweepWhileNotTimedOut() @bci=59, line=218 (Compiled frame)
 - c.e.s.w.d.CUQ$QueueExhauster.run() @bci=247, line=122 (Compiled frame)
 - c.e.s.w.d.CUQ$DequeuerStartFailed.run() @bci=40, line=60 (Compiled frame)
 - c.e.s.w.s.ew.CC.lambda$4(java.lang.Runnable) @bci=13, line=66 (Compiled frame)
 - c.e.s.w.s.ew.CC$$Lambda$59.run() @bci=8 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5 (Interpreted frame)
 - java.lang.Thread.run() @bci=11 (Compiled frame)

Thread 108: (state = BLOCKED) [The tricky one...]
 - java.lang.Thread.dumpThreads(java.lang.Thread[]) @bci=0 (Interpreted frame)
 - java.lang.Thread.getStackTrace() @bci=41 (Compiled frame)
 - c.e.s.w.SWA$$Lambda$98.getStackTrace() @bci=4 (Interpreted frame)
 - c.e.s.w.SWA.describeHoggingCode(c.e.s.w.s.RequestTimeTracker$StackTraceProvider, boolean) @bci=1, line=401 (Interpreted frame)
 - c.e.s.w.SWA.describeHoggingCode(c.e.s.w.s.RequestTimeTracker$StackTraceProvider, boolean, java.lang.Thread) @bci=6, line=396 (Interpreted frame)
 - c.e.s.w.SWA.lambda$10(java.lang.Thread, java.lang.String) @bci=8, line=890 (Interpreted frame)
 - c.e.s.w.SWA$$Lambda$62.run() @bci=12 (Interpreted frame)
 - c.e.s.w.s.ew.CC.lambda$4(java.lang.Runnable) @bci=13, line=66 (Compiled frame)
 - c.e.s.w.s.ew.CC$$Lambda$59.run() @bci=8 (Compiled frame)
 - c.e.s.w.s.IS.lambda$4(java.util.concurrent.atomic.AtomicBoolean, java.lang.Runnable) @bci=8, line=327 (Compiled frame)
 - c.e.s.w.s.IS$$Lambda$60.run() @bci=8 (Compiled frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4 (Compiled frame)
 - java.util.concurrent.FutureTask.run() @bci=42 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask) @bci=1 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() @bci=30 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5 (Interpreted frame)
 - java.lang.Thread.run() @bci=11 (Interpreted frame)

从我的角度来看，此问题可能与 Unsafe.defineAnonymousClass无法处理对 java.lang.Thread.dumpThreads的正在进行的调用有关(而这又需要在JVM中实现 java.lang.Thread.getStackTrace)。关键是，由于 final或package修饰符，我无法扩展此过程中涉及的任何核心类(例如 Lookup， MethodHandleNatives等)以引入将阻止棘手的Unsafe的锁仍在通话 java.lang.Thread.dumpThreads时进行通话。另外，我怀疑引入这样的锁定也会使速度降低很多，因为，到处都是lambda。

有没有人遇到过类似的问题？你能帮忙解决吗？

谢谢!

当然，在堆栈跟踪中也有类似这样的线程，我省略了这些线程，因为我认为它们与这种情况无关。

Thread 154: (state = BLOCKED) [Many of these....]
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78 (Compiled frame)
 - org.eclipse.jetty.util.BlockingArrayQueue.poll(long, java.util.concurrent.TimeUnit) @bci=57, line=389 (Compiled frame)
 - org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll() @bci=12, line=516 (Compiled frame)
 - org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(org.eclipse.jetty.util.thread.QueuedThreadPool) @bci=1, line=47 (Compiled frame)
 - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=300, line=575 (Compiled frame)
 - java.lang.Thread.run() @bci=11 (Compiled frame)


Thread 153: (state = BLOCKED) [and of these...]
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=354 (Compiled frame)
 - java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44 (Interpreted frame)
 - java.util.concurrent.ForkJoinWorkerThread.run() @bci=24 (Interpreted frame)


Thread 141: (state = BLOCKED) [and of these...]
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11 (Interpreted frame)

Answer 1

最终，我设法解决了这个令人讨厌的问题。如果将选项-XX:-ClassUnloading传递给JVM可执行文件(就像我尝试解决的JVM的情况)，那么JVM似乎会卡住类似被张贴(至少涉及sun.misc.Unsafe.defineAnonymousClass)的跟踪。由于某种原因，这可能会刺激Java系统库中的这种死锁样式的错误，该错误似乎是随机的(与所有死锁错误一样)，并且随着JVM进程“老化”而变得越来越可能。同样，对于死锁错误而言，这并不是什么新鲜事:将更多的“齿轮”(例如，本应卸载但不是由于该选项导致的类)放入机器中，则它们中的两个或两个以上发生曲柄的可能性就越大。

结果，删除选项-XX:-ClassUnloading使此问题完全消失。

最重要的是:切勿在生产系统中使用-XX:-ClassUnloading(或让任何人在JVM进程启动脚本中放置此类选项)，甚至在Java8, which shouldn't be limited by PermGen中也不要使用，但由于sun.misc.Unsafe.defineAnonymousClass的问题，它仍然可能遭受严重破坏。