python - TensorFlow v2 : CancelledError: [Op:StatefulPartitionedCall]

Question

在使用 TensorFlow v2 模型并尝试训练它们时，我不断收到此错误:

CancelledError: [Op:StatefulPartitionedCall]

我在 jupyter 笔记本上运行此程序，CPU 上装有最新的 beta 版 TensorFlow v2。

我的代码是:

def training_loop(model, data, num_epochs=10):

    def loss(x, y):
        error = tf.square(tf.subtract(model.predict(x), y))
        return error

    def grad(x, y):
        with tf.GradientTape() as tape:
            loss_value = loss(x, y)
        return loss_value, tape.gradient(loss_value, model.trainable_variables)

    learning_rate = 0.002
    optimizer = tf.train.AdamOptimizer(learning_rate)


    train_loss_results = []

    for epoch in range(num_epochs):

        epoch_loss_avg = tf.keras.metrics.Mean()

        # Training loop - using batches of 64
        for x, y in data:

            # Optimize the model
            loss_value, grads = grad(x.numpy(), y.numpy())
            optimizer.apply_gradients(zip(grads, model.trainable_variables))

        # Track progress
        epoch_loss_avg(loss_value)  # add current batch loss
        # compare predicted label to actual label

        # end epoch
        train_loss_results.append(epoch_loss_avg.result())

        if epoch % 50 == 0:
            print('Epoch {:03d}: Loss: {:.3f}'.format(epoch, epoch_loss_avg.result()))

当我尝试运行 training_loop 时，出现上述错误。

完整的回溯是:

CancelledError Traceback (most recent call last)
<ipython-input-49-57231385902d> in <module>
----> 1 p = ae.predict(x1)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in predict(self, x, batch_size, verbose, steps, max_queue_size, workers, use_multiprocessing)
1111 else:
1112 return training_arrays.predict_loop(
-> 1113 self, x, batch_size=batch_size, verbose=verbose, steps=steps)
1114
1115 def reset_metrics(self):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training_arrays.py in model_iteration(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps, mode, validation_in_fit, **kwargs)
327
328 # Get outputs.
--> 329 batch_outs = f(ins_batch)
330 if not isinstance(batch_outs, list):
331 batch_outs = [batch_outs]

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py in __call__(self, inputs)
3164 value = math_ops.cast(value, tensor.dtype)
3165 converted_inputs.append(value)
-> 3166 outputs = self._graph_fn(*converted_inputs)
3167 return nest.pack_sequence_as(self._outputs_structure,
3168 [x.numpy() for x in outputs])

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in __call__(self, *args, **kwargs)
366 raise TypeError("Got two values for keyword '{}'.".format(unused_key))
367 raise TypeError("Keyword arguments {} unknown.".format(kwargs.keys()))
--> 368 return self._call_flat(args)
369
370 def _filtered_call(self, args, kwargs):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _call_flat(self, args)
431 # Only need to override the gradient in graph mode and when we have outputs.
432 if context.executing_eagerly() or not self.outputs:
--> 433 outputs = self._inference_function.call(ctx, args)
434 else:
435 if not self._gradient_name:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in call(self, ctx, args)
267 executing_eagerly=executing_eagerly,
268 config=function_call_options.config_proto_serialized,
--> 269 executor_type=function_call_options.executor_type)
270
271 if executing_eagerly:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/functional_ops.py in partitioned_call(args, f, tout, executing_eagerly, config, executor_type)
1081 outputs = gen_functional_ops.stateful_partitioned_call(
1082 args=args, Tout=tout, f=f, config_proto=config,
-> 1083 executor_type=executor_type)
1084 else:
1085 outputs = gen_functional_ops.partitioned_call(

/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_functional_ops.py in stateful_partitioned_call(args, Tout, f, config, config_proto, executor_type, name)
495 else:
496 message = e.message
--> 497 _six.raise_from(_core._status_to_exception(e.code, message), None)
498 # Add nodes to the TensorFlow graph.
499 if not isinstance(Tout, (list, tuple)):

/usr/lib/python3/dist-packages/six.py in raise_from(value, from_value)

Answer 1

取消操作或步骤时，会引发 CancelledError，如您所见 here 。

实际提出的解决方案here为我工作:我刚刚用 model(x) 替换了 model.predict(x) 。我仍然不知道为什么这有效。然而，我的情况涉及一个模型，在该模型中我进行了带有手动定义梯度的自定义操作。据我从类似 this 的评论中可以理解和 this ，梯度的变化可能与Op:StatefulPartitionedCall有关。

请注意，CancelledError 似乎在笔记本中持续存在。一旦你得到它，你需要重新启动内核以使tensorflow再次正常工作。