python - 管理任务的 Google 应用引擎数据存储超时

Question

我为我的 Google 应用程序创建了一个后端，如下所示:

backends:
- name: dbops
  options: dynamic

我已经为其创建了一个管理处理程序:

- url: /backend/.*
  script: backend.app
  login: admin

现在我明白管理作业应该能够永远运行，并且我正在使用任务队列启动此作业，但由于某种原因我的作业不能。我的工作只是从一个更大的表在数据存储中创建一个汇总表。该表包含大约 12000 条记录，在开发服务器上处理该作业需要几分钟的时间，但它运行良好。当我将代码推送到 appspot 并尝试让它运行相同的作业时，我遇到了看起来像数据存储超时的情况。

Traceback (most recent call last):
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1536, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1530, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~myzencoder/dbops.362541511260492787/backend.py", line 626, in get
    for asset in assets:
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 2314, in next
    return self.__model_class.from_entity(self.__iterator.next())
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2816, in next
    next_batch = self.__batcher.next()
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2678, in next
    return self.next_batch(self.AT_LEAST_ONE)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2715, in next_batch
    batch = self.__next_batch.get_result()
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 604, in get_result
    return self.__get_result_hook(self)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2452, in __query_result_hook
    self._batch_shared.conn.check_rpc_success(rpc)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1224, in check_rpc_success
    raise _ToDatastoreError(err)
Timeout: The datastore operation timed out, or the data was temporarily unavailable.

有人对如何实现这项工作有任何建议吗？

Answer 1

虽然后端请求可以运行很长时间，但查询只能运行 60 秒。您必须使用游标循环查询结果。

Mapreduce 将通过并行执行查询更快地为您提供结果。