numpy - 5428 步后对象检测崩溃，类型错误 : 'numpy.float64' object cannot be interpreted as an integer

Question

我的对象检测器已运行多次，但在 5428 的标记处，它随后因 TypeError 的错误而崩溃

我在 anaconda 中运行:

numpy 1.18.1

numpy-base 1.18.1

tensorflow -GPU 1.14

下面这个片段我认为是最重要的错误？:

2020-02-19 13:56:06.901096: W tensorflow/core/framework/op_kernel.cc:1490] Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
Traceback (most recent call last):

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\numpy\core\function_base.py", line 117, in linspace
    num = operator.index(num)

TypeError: 'numpy.float64' object cannot be interpreted as an integer

完整的回溯如下:

I0219 13:55:41.016854 15428 basic_session_run_hooks.py:260] loss = 0.0140173, step = 5400 (10.773 sec)
INFO:tensorflow:Saving checkpoints for 5428 into training/model.ckpt.
I0219 13:55:43.900022 15428 basic_session_run_hooks.py:606] Saving checkpoints for 5428 into training/model.ckpt.
INFO:tensorflow:Calling model_fn.
I0219 13:55:56.207441 15428 estimator.py:1145] Calling model_fn.
INFO:tensorflow:Scale of 0 disables regularizer.
I0219 13:55:58.009801 15428 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0219 13:55:58.025418 15428 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0219 13:55:58.025418 15428 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
I0219 13:55:59.573186 15428 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0219 13:55:59.588815 15428 regularizers.py:98] Scale of 0 disables regularizer.
WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\eval_util.py:796: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0219 13:56:00.855241 15428 deprecation.py:323] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\eval_util.py:796: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\utils\visualization_utils.py:498: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.

W0219 13:56:01.105266 15428 deprecation.py:323] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\utils\visualization_utils.py:498: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.

WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\utils\visualization_utils.py:1044: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.

W0219 13:56:01.277014 15428 deprecation_wrapper.py:119] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\utils\visualization_utils.py:1044: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.

WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\model_lib.py:484: The name tf.metrics.mean is deprecated. Please use tf.compat.v1.metrics.mean instead.

W0219 13:56:01.386395 15428 deprecation_wrapper.py:119] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\model_lib.py:484: The name tf.metrics.mean is deprecated. Please use tf.compat.v1.metrics.mean instead.

INFO:tensorflow:Done calling model_fn.
I0219 13:56:01.749697 15428 estimator.py:1147] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-02-19T13:56:01Z
I0219 13:56:01.781106 15428 evaluation.py:255] Starting evaluation at 2020-02-19T13:56:01Z
INFO:tensorflow:Graph was finalized.
I0219 13:56:02.489665 15428 monitored_session.py:240] Graph was finalized.
2020-02-19 13:56:02.508162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:06:00.0
2020-02-19 13:56:02.512995: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-02-19 13:56:02.516493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-02-19 13:56:02.518703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-19 13:56:02.523922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2020-02-19 13:56:02.526614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2020-02-19 13:56:02.529223: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8788 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:06:00.0, compute capability: 6.1)
WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0219 13:56:02.535778 15428 deprecation.py:323] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from training/model.ckpt-5428
I0219 13:56:02.538779 15428 saver.py:1280] Restoring parameters from training/model.ckpt-5428
INFO:tensorflow:Running local_init_op.
I0219 13:56:03.495252 15428 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0219 13:56:03.656017 15428 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 5 images.
I0219 13:56:06.852077 13368 coco_evaluation.py:205] Performing evaluation on 5 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0219 13:56:06.867704 13368 coco_tools.py:115] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.00s)
I0219 13:56:06.867704 13368 coco_tools.py:137] DONE (t=0.00s)
creating index...
index created!
2020-02-19 13:56:06.901096: W tensorflow/core/framework/op_kernel.cc:1490] Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
Traceback (most recent call last):

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\numpy\core\function_base.py", line 117, in linspace
    num = operator.index(num)

TypeError: 'numpy.float64' object cannot be interpreted as an integer


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 209, in __call__
    ret = func(*args)

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_evaluation.py", line 384, in first_value_func
    self._metrics = self.evaluate()

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_evaluation.py", line 215, in evaluate
    coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False)

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_tools.py", line 176, in __init__
    iouType=iou_type)

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 76, in __init__
    self.params = Params(iouType=iouType) # parameters

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 527, in __init__
    self.setDetParams()

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 507, in setDetParams
    self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)

  File "<__array_function__ internals>", line 6, in linspace

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\numpy\core\function_base.py", line 121, in linspace
    .format(type(num)))

TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.


Traceback (most recent call last):
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1356, in _do_call
    return fn(*args)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
         [[{{node IteratorGetNext}}]]
  (1) Out of range: End of sequence
         [[{{node IteratorGetNext}}]]
         [[Loss/BoxClassifierLoss/assert_equal/Assert/Assert/data_4/_2449]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\evaluation.py", line 272, in _evaluate_once
    session.run(eval_ops, feed_dict)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1252, in run
    run_metadata=run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1353, in run
    raise six.reraise(*original_exc_info)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\six.py", line 703, in reraise
    raise value
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1338, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1411, in run
    run_metadata=run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1169, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 950, in run
    run_metadata_ptr)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_run
    run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
         [[node IteratorGetNext (defined at model_main.py:105) ]]
  (1) Out of range: End of sequence
         [[node IteratorGetNext (defined at model_main.py:105) ]]
         [[Loss/BoxClassifierLoss/assert_equal/Assert/Assert/data_4/_2449]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'IteratorGetNext':
  File "model_main.py", line 109, in <module>
    tf.app.run()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\absl\app.py", line 299, in run
    _run_main(main, args)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\absl\app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473, in train_and_evaluate
    return executor.run()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613, in run
    return self.run_local()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 367, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1158, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1192, in _train_model_default
    saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1484, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1252, in run
    run_metadata=run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1338, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1419, in run
    run_metadata=run_metadata))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 594, in after_run
    if self._save(run_context.session, global_step):
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 619, in _save
    if l.after_save(session, step):
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 519, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 539, in _evaluate
    self._evaluator.evaluate_and_export())
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 920, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 477, in evaluate
    name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 519, in _actual_eval
    return _evaluate()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 501, in _evaluate
    self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1501, in _evaluate_build_graph
    self._call_model_fn_eval(input_fn, self.config))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1534, in _call_model_fn_eval
    input_fn, ModeKeys.EVAL)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1022, in _get_features_and_labels_from_input_fn
    self._call_input_fn(input_fn, mode))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\util.py", line 65, in parse_input_fn_result
    result = iterator.get_next()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 426, in get_next
    output_shapes=self._structure._flat_shapes, name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 1947, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3616, in create_op
    op_def=op_def)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()



During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 209, in __call__
    ret = func(*args)

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_evaluation.py", line 384, in first_value_func
    self._metrics = self.evaluate()

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_evaluation.py", line 215, in evaluate
    coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False)

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_tools.py", line 176, in __init__
    iouType=iou_type)

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 76, in __init__
    self.params = Params(iouType=iouType) # parameters

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 527, in __init__
    self.setDetParams()

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 507, in setDetParams
    self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)

  File "<__array_function__ internals>", line 6, in linspace

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\numpy\core\function_base.py", line 121, in linspace
    .format(type(num)))

TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.


         [[node PyFunc_3 (defined at C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_evaluation.py:394) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'PyFunc_3':
  File "model_main.py", line 109, in <module>
    tf.app.run()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\absl\app.py", line 299, in run
    _run_main(main, args)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\absl\app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473, in train_and_evaluate
    return executor.run()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613, in run
    return self.run_local()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 367, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1158, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1192, in _train_model_default
    saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1484, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1252, in run
    run_metadata=run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1338, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1419, in run
    run_metadata=run_metadata))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 594, in after_run
    if self._save(run_context.session, global_step):
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 619, in _save
    if l.after_save(session, step):
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 519, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 539, in _evaluate
    self._evaluator.evaluate_and_export())
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 920, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 477, in evaluate
    name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 519, in _actual_eval
    return _evaluate()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 501, in _evaluate
    self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1501, in _evaluate_build_graph
    self._call_model_fn_eval(input_fn, self.config))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1537, in _call_model_fn_eval
    features, labels, ModeKeys.EVAL, config)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1146, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\model_lib.py", line 482, in model_fn
    eval_config, list(category_index.values()), eval_dict)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\eval_util.py", line 947, in get_eval_metric_ops_for_evaluators
    eval_dict))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_evaluation.py", line 394, in get_estimator_eval_metric_ops
    first_value_op = tf.py_func(first_value_func, [], tf.float32)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 480, in py_func
    return py_func_common(func, inp, Tout, stateful, name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 462, in py_func_common
    func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 285, in _internal_py_func
    input=inp, token=token, Tout=Tout, name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\gen_script_ops.py", line 159, in py_func
    "PyFunc", input=input, token=token, Tout=Tout, name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3616, in create_op
    op_def=op_def)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

由于字符限制，这缺少一些回溯，但它都是 numpy TypeError 的。

Answer 1

尝试降级您的 numpy 版本。
就我而言，我不得不将其降级到 1.17.4