python - Google对象检测API——使用faster_rcnn_resnet101_coco模型进行训练

Question

我使用 mobilenet 模型来训练我的图像。它运作良好。为了提高准确性，我尝试使用 faster_rcnn_resnet101_coco 模型来复制相同的步骤。我使用的所有步骤都是相同的。当我开始训练类(class)时，它开始并跑了大约 800 步。此时的训练损失约为 0.5，这似乎好得令人难以置信。它在这一步停止并抛出以下错误:

The replica worker 1 exited with a non-zero status of 1. Termination reason: Error. Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/root/.local/lib/python2.7/site-packages/object_detection/train.py", line 163, in tf.app.run() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "/root/.local/lib/python2.7/site-packages/object_detection/train.py", line 159, in main worker_job_name, is_chief, FLAGS.train_dir) File "/root/.local/lib/python2.7/site-packages/object_detection/trainer.py", line 332, in train saver=saver) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 763, in train sess, train_op, global_step, train_step_kwargs) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 487, in train_step run_metadata=run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 889, in run run_metadata_ptr) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1120, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1317, in _do_run options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1336, in _do_call raise type(e)(node_def, op, message) UnavailableError: Endpoint read failed To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=341450659208&resource=ml_job%2Fjob_id%2Fobject_detection_188003&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22object_detection_188003%22

知道问题出在哪里吗？非常感谢任何帮助。

Answer 1

感谢您的反馈。我们仍在调查此问题，请暂时使用 1.2 运行时版本。