gpt4 book ai didi

python - 应用程序错误 : 7 when processing is done for mapreduce worker writing to Google Cloud Storage

转载 作者:太空狗 更新时间:2023-10-30 01:29:45 24 4
gpt4 key购买 nike

刚刚完成一个大型的 Appengine mapreduce 任务,我的许多分片都卡在了终点线上。这是设置:

    filenames = yield mapreduce_pipeline.MapperPipeline(
'example mapper name',
'main.MyMapper',
input_reader_spec='mapreduce.input_readers.DatastoreInputReader',
output_writer_spec='mapreduce.output_writers.FileOutputWriter',
params={
'input_reader':{
'entity_kind':'models.MyModel'
},
'output_writer':{
'filesystem':'gs',
'mime_type':'text/csv',
'gs_bucket_name':'myBucket',
'output_sharding':'input'
}
},
shards=DUMP_SHARDS
)

我正在并行运行其中的 3 个,每个都有 16 个分片。一个映射器顺利完成,另外两个映射器在 14 个和 9 个分片上都取得了成功。

剩下的分片都被完全阻止了,返回UnknownError: ApplicationError: 7。 (本文末尾的完整堆栈跟踪。)

请注意,映射器正在尝试写入 Google Cloud Storage。错误发生在执行此写入的位中。

找了一段时间后,我发现,在google.appengine.runtime.apiproxy (这似乎是有问题的代理),错误 7 是 OTHER_ERROR

我已经重试了这些最终任务(从任务队列中)大约 3 个小时,而且自从这些错误开始以来没有一个成功;无论发生什么,它都完全卡住了。我也试过停止所有正在运行的实例,以防它是一些奇怪的本地状态,但那里没有变化......

这是完整的堆栈跟踪:

I 2012-12-13 15:40:23.909
Processing done for shard 14 of job '1582444192075C233F6AA'
E 2012-12-13 15:40:23.969
ApplicationError: 7
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/base_handler.py", line 65, in post
self.handle()
File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/handlers.py", line 231, in handle
tstate.output_writer.finalize(ctx, shard_state.shard_number)
File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/output_writers.py", line 631, in finalize
files.finalize(self._filename)
File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/lib/files/file.py", line 568, in finalize
f.close(finalize=True)
File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/lib/files/file.py", line 291, in close
self._make_rpc_call_with_retry('Close', request, response)
File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/lib/files/file.py", line 427, in _make_rpc_call_with_retry
_make_call(method, request, response)
File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/lib/files/file.py", line 252, in _make_call
_raise_app_error(e)
File "/base/data/home/apps/myserver/myinstance.363844686987482417/mapreduce/lib/files/file.py", line 186, in _raise_app_error
raise UnknownError(e)
UnknownError: ApplicationError: 7

最佳答案

我刚刚遇到了类似的问题。我认为这特别是写入 Google Cloud Storage 的问题。

我在这里获得了一些见解:Google App Engine Issue: 8775

总结(TLDR):

  • 可能是一次性网络问题。
  • 可能是计费问题。
  • 结果:如果问题没有消失并且修复结算也不起作用,请联系 Google 支持。

关于python - 应用程序错误 : 7 when processing is done for mapreduce worker writing to Google Cloud Storage,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/13870975/

24 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com