tensorflow2.0 - 使用 TF2 微调大型通用句子编码器

Question

下面是我用于微调 Universal Sentence Encoder Multilingual Large 2 的代码。我无法解决由此产生的错误。我尝试添加一个 tf.keras.layers.Input 层，这导致了同样的错误。非常感谢任何关于如何为 USEM2 成功构建微调序列模型的建议。

import tensorflow as tf
import tensorflow_text
import tensorflow_hub as hub

module_url = "https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/2"

embedding_layer = hub.KerasLayer(module_url, trainable=True, input_shape=[None,], dtype=tf.string)
hidden_layer = tf.keras.layers.Dense(32, activation='relu')
output_layer = tf.keras.layers.Dense(5, activation='softmax')

model = tf.keras.models.Sequential()

model.add(embedding_layer)
model.add(hidden_layer)
model.add(output_layer)

model.summary()

WARNING:tensorflow:Entity <tensorflow.python.saved_model.function_deserialization.RestoredFunction object at 0x7fdf34216390> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Shape must be rank 1 but is rank 2 for 'text_preprocessor_1/SentenceTokenizer/SentencepieceTokenizeOp' (op: 'SentencepieceTokenizeOp') with input shapes: [], [?,?], [], [], [], [], [].

WARNING:tensorflow:Entity <tensorflow.python.saved_model.function_deserialization.RestoredFunction object at 0x7fdf34216390> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Shape must be rank 1 but is rank 2 for 'text_preprocessor_1/SentenceTokenizer/SentencepieceTokenizeOp' (op: 'SentencepieceTokenizeOp') with input shapes: [], [?,?], [], [], [], [], [].

WARNING: Entity <tensorflow.python.saved_model.function_deserialization.RestoredFunction object at 0x7fdf34216390> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Shape must be rank 1 but is rank 2 for 'text_preprocessor_1/SentenceTokenizer/SentencepieceTokenizeOp' (op: 'SentencepieceTokenizeOp') with input shapes: [], [?,?], [], [], [], [], [].

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-61-7ea0d071abf8> in <module>
      1 model = tf.keras.models.Sequential()
      2 
----> 3 model.add(embedding_layer)
      4 model.add(hidden_layer)
      5 model.add(output)

~/pyenv36/lib/python3.6/site-packages/tensorflow_core/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
    455     self._self_setattr_tracking = False  # pylint: disable=protected-access
    456     try:
--> 457       result = method(self, *args, **kwargs)
    458     finally:
    459       self._self_setattr_tracking = previous_value  # pylint: disable=protected-access

~/pyenv36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/sequential.py in add(self, layer)
    176           # and create the node connecting the current layer
    177           # to the input layer we just created.
--> 178           layer(x)
    179           set_inputs = True
    180 

~/pyenv36/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    840                     not base_layer_utils.is_in_eager_or_tf_function()):
    841                   with auto_control_deps.AutomaticControlDependencies() as acd:
--> 842                     outputs = call_fn(cast_inputs, *args, **kwargs)
    843                     # Wrap Tensors in `outputs` in `tf.identity` to avoid
    844                     # circular dependencies.

~/pyenv36/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    235       except Exception as e:  # pylint:disable=broad-except
    236         if hasattr(e, 'ag_error_metadata'):
--> 237           raise e.ag_error_metadata.to_exception(e)
    238         else:
    239           raise

ValueError: in converted code:
    relative to /home/neubig/pyenv36/lib/python3.6/site-packages:

    tensorflow_hub/keras_layer.py:209 call  *
        result = f()
    tensorflow_core/python/saved_model/load.py:436 _call_attribute
        return instance.__call__(*args, **kwargs)
    tensorflow_core/python/eager/def_function.py:457 __call__
        result = self._call(*args, **kwds)
    tensorflow_core/python/eager/def_function.py:494 _call
        results = self._stateful_fn(*args, **kwds)
    tensorflow_core/python/eager/function.py:1823 __call__
        return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
    tensorflow_core/python/eager/function.py:1141 _filtered_call
        self.captured_inputs)
    tensorflow_core/python/eager/function.py:1230 _call_flat
        flat_outputs = forward_function.call(ctx, args)
    tensorflow_core/python/eager/function.py:540 call
        executor_type=executor_type)
    tensorflow_core/python/ops/functional_ops.py:859 partitioned_call
        executor_type=executor_type)
    tensorflow_core/python/ops/gen_functional_ops.py:672 stateful_partitioned_call
        executor_type=executor_type, name=name)
    tensorflow_core/python/framework/op_def_library.py:793 _apply_op_helper
        op_def=op_def)
    tensorflow_core/python/framework/func_graph.py:548 create_op
        compute_device)
    tensorflow_core/python/framework/ops.py:3429 _create_op_internal
        op_def=op_def)
    tensorflow_core/python/framework/ops.py:1773 __init__
        control_input_ops)
    tensorflow_core/python/framework/ops.py:1613 _create_c_op
        raise ValueError(str(e))

    ValueError: Shape must be rank 1 but is rank 2 for 'text_preprocessor_1/SentenceTokenizer/SentencepieceTokenizeOp' (op: 'SentencepieceTokenizeOp') with input shapes: [], [?,?], [], [], [], [], [].

Answer 1

据我所知，目前 tf.hub 中的 Universal Sentence Encoder Multilingual 不支持 trainable=True。

但是，这些代码片段可以使模型进行推理:

使用 V2

module_url = "https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/2"
embedding_layer = hub.KerasLayer(module_url)
hidden_layer = tf.keras.layers.Dense(32, activation='relu')
output_layer = tf.keras.layers.Dense(5, activation='softmax')

inputs = tf.keras.layers.Input(shape=(1,), dtype=tf.string)
x = embedding_layer(tf.squeeze(tf.cast(inputs, tf.string)))["outputs"]
x = hidden_layer(x)
outputs = output_layer(x)

model = tf.keras.Model(inputs=inputs, outputs=outputs)

使用 V3

module_url = "https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/3"
embedding_layer = hub.KerasLayer(module_url)
hidden_layer = tf.keras.layers.Dense(32, activation='relu')
output_layer = tf.keras.layers.Dense(5, activation='softmax')

inputs = tf.keras.layers.Input(shape=(1,), dtype=tf.string)
x = embedding_layer(tf.squeeze(tf.cast(inputs, tf.string)))
x = hidden_layer(x)
outputs = output_layer(x)

model = tf.keras.Model(inputs=inputs, outputs=outputs)

推理

model.predict([["hello tf2"]])