java - 处理 HardDeadlineExceededError 的任务策略

Question

我有许多任务/servlet 遇到了 HardDeadlineExceededError，这使所有内容都处于“仍在执行”状态。

正在完成的工作很容易超过 29 秒的阈值。

我 try catch DeadlineExceededException 和基本异常以保存退出状态，但这些异常处理程序都没有被捕获...<​​/p>

有没有办法确定哪些任务在队列中或当前正在执行？

是否有其他策略来处理这种情况？

我正在处理的情况记录在 "The Request Timer" 下标题。

// task handler for retrieving information from external web services
protected void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {

    String taskRetryCountParam = req.getParameter( "X-AppEngine-TaskRetryCount" );
    int taskRetryCount = (taskRetryCountParam==null) ? 0 : Integer.parseInt(taskRetryCountParam);
            // look up the persistent 'task' and mark it as 'running'

    logger.info( this.getClass().getName() + ".doPost( "+ taskId + " ) retryCount=" + taskRestryCount );


    // Do lots of heavy lifting here 
    // like calling external web services using URL fetch service
            // and saving the contents into our database.

            // look up the persistent 'task' and mark it as 'completed'

    } catch ( DeadlineExceededException deadline ) {
        // got this deadline exception
                    // look up the persistent 'task' and mark it as 'errored - try again'
        logger.warning( "DeadlineExceeded Exception while loading content " + deadline.getMessage() );
        resp.setStatus( HttpServletResponse.SC_REQUEST_TIMEOUT );

        }
    } catch ( Exception unknown ) {
        // got some unknown exception
                    // look up the persistent 'task' and mark it as 'errored - cancelled'
        logger.severe( "General Exception while loading content  exception:" + unknown.getMessage() );
        resp.setStatus( HttpServletResponse.SC_OK );

    }
}

---

这是我遇到这种情况时的日志文件条目...看来我的数据库事务在需要时花费的时间太长了。

  W 05-30 12:42PM 09.535
  Error for /loadstatus
  com.google.apphosting.runtime.HardDeadlineExceededError: This request (083793d1091c2ca3) started at 2010/05/30 19:41:39.814 UTC and was still executing at 2010/05/30 19:42:09.529 UTC.
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:443)
    at java.util.concurrent.TimeUnit.timedWait(Unknown Source)
    at com.google.apphosting.runtime.AsyncFuture.get(AsyncFuture.java:60)
    at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.get(ApiProxyImpl.java:326)
    at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.get(ApiProxyImpl.java:217)
    at com.google.apphosting.runtime.ApiProxyImpl.doSyncCall(ApiProxyImpl.java:131)
    at com.google.apphosting.runtime.ApiProxyImpl.access$000(ApiProxyImpl.java:43)
    at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:104)
    at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:102)
    at java.security.AccessController.doPrivileged(Native Method)
    at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:102)
    at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:43)
    at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:98)
    at com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(DatastoreApiHelper.java:58)
    at com.google.appengine.api.datastore.TransactionImpl.makeSyncCall(TransactionImpl.java:42)
    at com.google.appengine.api.datastore.TransactionImpl.makeSyncCall(TransactionImpl.java:56)
    at com.google.appengine.api.datastore.TransactionImpl.commit(TransactionImpl.java:66)
    at org.datanucleus.store.appengine.DatastoreTransaction.commit(DatastoreTransaction.java:61)
    at org.datanucleus.store.appengine.DatastoreXAResource.commit(DatastoreXAResource.java:88)
    at org.datanucleus.transaction.Transaction.commit(Transaction.java:149)
    at org.datanucleus.transaction.TransactionManager.commit(TransactionManager.java:95)
    at org.datanucleus.TransactionImpl.internalCommit(TransactionImpl.java:390)
    at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:258)
    at org.datanucleus.jdo.JDOTransaction.commit(JDOTransaction.java:83)
    at org.datanucleus.store.appengine.jdo.DatastoreJDOTransaction.commit(DatastoreJDOTransaction.java:56)
    <snip>
  C 05-30 12:42PM 09.629
  Uncaught exception from servlet
  com.google.apphosting.runtime.HardDeadlineExceededError: This request (083793d1091c2ca3) started at 2010/05/30 19:41:39.814 UTC and was still executing at 2010/05/30 19:42:09.529 UTC.
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:443)
    at java.util.concurrent.TimeUnit.timedWait(Unknown Source)
    at com.google.apphosting.runtime.AsyncFuture.get(AsyncFuture.java:60)
    at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.get(ApiProxyImpl.java:326)
    at com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.get(ApiProxyImpl.java:217)
    at com.google.apphosting.runtime.ApiProxyImpl.doSyncCall(ApiProxyImpl.java:131)
    at com.google.apphosting.runtime.ApiProxyImpl.access$000(ApiProxyImpl.java:43)
    at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:104)
    at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:102)
    at java.security.AccessController.doPrivileged(Native Method)
    at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:102)
    at com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:43)
    at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:98)
    at com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(DatastoreApiHelper.java:58)
    at com.google.appengine.api.datastore.TransactionImpl.makeSyncCall(TransactionImpl.java:42)
    at com.google.appengine.api.datastore.TransactionImpl.makeSyncCall(TransactionImpl.java:56)
    at com.google.appengine.api.datastore.TransactionImpl.commit(TransactionImpl.java:66)
    at org.datanucleus.store.appengine.DatastoreTransaction.commit(DatastoreTransaction.java:61)
    at org.datanucleus.store.appengine.DatastoreXAResource.commit(DatastoreXAResource.java:88)
    at org.datanucleus.transaction.Transaction.commit(Transaction.java:149)
    at org.datanucleus.transaction.TransactionManager.commit(TransactionManager.java:95)
    at org.datanucleus.TransactionImpl.internalCommit(TransactionImpl.java:390)
    at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:258)
    at org.datanucleus.jdo.JDOTransaction.commit(JDOTransaction.java:83)
    at org.datanucleus.store.appengine.jdo.DatastoreJDOTransaction.commit(DatastoreJDOTransaction.java:56)
    <snip>
  W 05-30 12:42PM 09.644
  A serious problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104)

Answer 1

http://groups.google.com/group/google-appengine-java/msg/e3fd2b621bb96013

HDEEs can be thrown without a DEE if it happens in your own code.
Usually the most time consuming thing is waiting for API calls to
return so a timeout here will result in the API call stopping with a
DEE. So if you do not call the API often you can hit HDEE directly.

I also have long running tasks that iterate through data processing
and storing results. I use an Iterator that stops returning results
after 20 seconds and saves the last object processed and then kicks
off a new task to continue processing.

My original solution caught the DEE and then cleaned up but this
stopped working reliably.