java - 实例因死锁而崩溃

Question

我们再次遇到实例崩溃，这一次使用 Samurai 工具，我发现几个区域出现了死锁。我们在死锁中发现的代码片段之一如下。相应的应用程序代码在第 1 行引用。 898 只是一个简单的语句，它试图用 catch block 关闭一个语句。不知道为什么这会导致死锁。需要帮助来解决此问题。

    at weblogic.jdbc.wrapper.Statement.close(Statement.java:293)
    at com.crimsonlogic.trade.sdec.sau.util.CommonUtil.getHijri(CommonUtil.java:898)
    at com.crimsonlogic.trade.sdec.sau.model.SAUSDECVehicleTransaction.setHijrifromGregorian_trnDate(SAUSDECVehicleTransaction.java:261)
    at com.crimsonlogic.trade.sdec.sau.dao.SAUSDECVehicleTransactionDAOImpl.search(SAUSDECVehicleTransactionDAOImpl.java:175)
    at com.crimsonlogic.trade.sdec.sau.dao.SAUSDECMasterDAOImpl.searchSDECMsg(SAUSDECMasterDAOImpl.java:310)
    at com.crimsonlogic.trade.sdec.sau.ejb.SAUSDECSvcBean.search(SAUSDECSvcBean.java:452)
    at com.crimsonlogic.trade.sdec.sau.ejb.SAUSDECSvcBean_bxozws_EOImpl.search(SAUSDECSvcBean_bxozws_EOImpl.java:1290)



Found one Java-level deadlock:
=============================
"asyncDelivery42":
  waiting to lock monitor 0xdfb820 (object 0x87351698, a oracle.jdbc.driver.OracleConnection),
  which is held by "ExecuteThread: '93' for queue: 'weblogic.kernel.Default'"
"ExecuteThread: '93' for queue: 'weblogic.kernel.Default'":
  waiting to lock monitor 0xdbf550 (object 0x873782b0, a oracle.jdbc.driver.OracleCallableStatement),
  which is held by "asyncDelivery42"

Java stack information for the threads listed above:
===================================================
"asyncDelivery42":
        at oracle.jdbc.driver.OracleConnection.getDefaultRowPrefetch(OracleConnection.java:1866)
        - waiting to lock <87351698> (a oracle.jdbc.driver.OracleConnection)
        at oracle.jdbc.driver.OracleStatement.setPrefetchInternal(OracleStatement.java:1635)
        at oracle.jdbc.driver.OracleStatement.setFetchSize(OracleStatement.java:6505)
        - locked <873782b0> (a oracle.jdbc.driver.OracleCallableStatement)
        at weblogic.jdbc.common.internal.ConnectionEnv.cleanUpStatementForReUse(ConnectionEnv.java:1091)
        at weblogic.jdbc.common.internal.ConnectionEnv.returnCachedStatement(ConnectionEnv.java:883)
        at weblogic.jdbc.wrapper.Statement.internalClose(Statement.java:271)
        at weblogic.jdbc.wrapper.Statement.close(Statement.java:293)
        at com.crimsonlogic.trade.sdec.sau.util.CommonUtil.getHijri(CommonUtil.java:898)
        at com.crimsonlogic.trade.sdec.sau.model.SAUSDECVehicleTransaction.setHijrifromGregorian_trnDate(SAUSDECVehicleTransaction.java:261)
        at com.crimsonlogic.trade.sdec.sau.dao.SAUSDECVehicleTransactionDAOImpl.search(SAUSDECVehicleTransactionDAOImpl.java:175)
        at com.crimsonlogic.trade.sdec.sau.dao.SAUSDECMasterDAOImpl.searchSDECMsg(SAUSDECMasterDAOImpl.java:310)
        at com.crimsonlogic.trade.sdec.sau.ejb.SAUSDECSvcBean.search(SAUSDECSvcBean.java:452)
        at com.crimsonlogic.trade.sdec.sau.ejb.SAUSDECSvcBean_bxozws_EOImpl.search(SAUSDECSvcBean_bxozws_EOImpl.java:1290)
        at com.crimsonlogic.trade.sdec.sau.ejb.SAUSDECSvcBean_bxozws_EOImpl_CBV.search(Unknown Source)
        at com.crimsonlogic.trade.sdec.sau.business.SAUSDECDelegate.search(SAUSDECDelegate.java:278)
        at com.crimsonlogic.trade.sdec.sau.business.SDECMsgProcessor.process(SDECMsgProcessor.java:72)
        at com.crimsonlogic.trade.sau.msg.ejb.MsgSvcBean.processMsg(MsgSvcBean.java:201)
        at com.crimsonlogic.trade.sau.msg.ejb.MsgSvcBean_5z2zq6_EOImpl.processMsg(MsgSvcBean_5z2zq6_EOImpl.java:1340)
        at com.crimsonlogic.trade.sau.msg.util.MsgDelegate.processMsg(MsgDelegate.java:196)
        at com.crimsonlogic.trade.sau.msg.ejb.ProcessPortMsgIDMDB.onMessage(ProcessPortMsgIDMDB.java:72)
        at weblogic.ejb20.internal.MDListener.execute(MDListener.java:382)
        at weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:316)
        at weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:281)
        at com.ibm.mq.jms.MQMessageConsumer.receiveAsyncQ(MQMessageConsumer.java:2431)
        - locked <7bc2f570> (a java.lang.Object)
        - locked <7bc2f4e8> (a com.ibm.mq.jms.MQQueueReceiver)
        at com.ibm.mq.jms.MQMessageConsumer.receiveAsync(MQMessageConsumer.java:3816)
        - locked <7bc2f4e8> (a com.ibm.mq.jms.MQQueueReceiver)
        at com.ibm.mq.jms.SessionAsyncHelper.run(SessionAsyncHelper.java:420)
        at java.lang.Thread.run(Thread.java:536)
"ExecuteThread: '93' for queue: 'weblogic.kernel.Default'":
        at oracle.jdbc.driver.OracleCallableStatement.sendBatch(OracleCallableStatement.java:446)
        - waiting to lock <873782b0> (a oracle.jdbc.driver.OracleCallableStatement)
        at oracle.jdbc.driver.OracleConnection.commit(OracleConnection.java:1336)
        - locked <87351698> (a oracle.jdbc.driver.OracleConnection)
        at weblogic.jdbc.wrapper.JTSConnection.internalCommit(JTSConnection.java:362)
        at weblogic.jdbc.wrapper.JTSXAResourceImpl.commit(JTSXAResourceImpl.java:56)
        at weblogic.transaction.internal.XAServerResourceInfo.commit(XAServerResourceInfo.java:1223)
        at weblogic.transaction.internal.XAServerResourceInfo.commit(XAServerResourceInfo.java:472)
        at weblogic.transaction.internal.ServerSCInfo.startCommit(ServerSCInfo.java:421)
        at weblogic.transaction.internal.ServerTransactionImpl.localCommit(ServerTransactionImpl.java:1761)
        at weblogic.transaction.internal.ServerTransactionImpl.globalRetryCommit(ServerTransactionImpl.java:2384)
        at weblogic.transaction.internal.ServerTransactionImpl.globalCommit(ServerTransactionImpl.java:2315)
        at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:255)
        at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:221)
        at weblogic.ejb20.internal.BaseEJBObject.postInvoke(BaseEJBObject.java:289)
        at weblogic.ejb20.internal.StatelessEJBObject.postInvoke(StatelessEJBObject.java:141)
        at com.crimsonlogic.trade.sau.msg.ejb.MsgSvcBean_5z2zq6_EOImpl.checkLinkedRefIDStatus(MsgSvcBean_5z2zq6_EOImpl.java:110)
        at com.crimsonlogic.trade.sau.msg.util.MsgDelegate.checkLinkedRefIDStatus(MsgDelegate.java:824)
        at com.crimsonlogic.trade.sau.msg.action.ProcessMsg.getProcessMsgIDList(ProcessMsg.java:260)
        at com.crimsonlogic.trade.sau.msg.action.ProcessMsg._process(ProcessMsg.java:123)
        at com.crimsonlogic.trade.sau.msg.action.AbstractAction.process(AbstractAction.java:61)
        at com.crimsonlogic.trade.sau.msg.servlet.SAUMsgProcessServlet.doPost(SAUMsgProcessServlet.java:102)
        at com.crimsonlogic.trade.sau.msg.servlet.SAUMsgProcessServlet.doGet(SAUMsgProcessServlet.java:52)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
        at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:971)
        at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:402)
        at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:305)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6354)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:317)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
        at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3635)
        at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2585)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

Found 2 deadlocks.

Answer 1

请检查 MsgDelegate 中的 checkLinkedRefIDStatus() 方法。它是从 SAUMsgProcessServlet(servlet 容器线程)和 ProcessPortMsgIDMDB.onMessage(MQ Series MDB 监听器线程)调用的。

我的猜测是，此方法始终使用相同的 JDBC 连接，而不是从池中借用/打开新连接。我眼前没有 Oracle JDBC 驱动程序源代码，但我敢打赌 oracle.jdbc.driver.OracleConnection.commit 已同步，并且显然有多个线程命中 checkLinkedRefIDStatus 正在共享同一连接。