python - 如何将以下 tf 1.x 代码转换为 tf 2.0(对现有代码的更改最少)

Question

我正在从 tensorflow 1.x 迁移代码至tensorflow-2.0 .我使用了tensorflow-2.0 中提供的转换脚本这很好。但是脚本无法转换 tf.contrib模块的代码。我想做下面的代码tensorflow-2.0兼容的。

def dropout(input_tensor, dropout_prob):
  """Perform dropout.

  Args:
    input_tensor: float Tensor.
    dropout_prob: Python float. The probability of dropping out a value (NOT of
      *keeping* a dimension as in `tf.nn.dropout`).

  Returns:
    A version of `input_tensor` with dropout applied.
  """
  if dropout_prob is None or dropout_prob == 0.0:
    return input_tensor

  output = tf.nn.dropout(input_tensor, 1 - (1.0 - dropout_prob))
  return output

def layer_norm(input_tensor, name=None):
  """Run layer normalization on the last dimension of the tensor."""
  return tf.contrib.layers.layer_norm(
      inputs=input_tensor, begin_norm_axis=-1, begin_params_axis=-1, scope=name)


def layer_norm_and_dropout(input_tensor, dropout_prob, name=None):
  """Runs layer normalization followed by dropout."""
  output_tensor = layer_norm(input_tensor, name)
  output_tensor = dropout(output_tensor, dropout_prob)
  return output_tensor

我遇到的错误:

1) 在弃用的模块 tf.contrib.layer_norm 中使用成员 tf.contrib.layers.layer_norm

我在互联网上的搜索找到了我这个 github issue

但是，如何迁移仍不清楚。

提前致谢。

Answer 1

对于层规范化，迁移到 Keras 层对我有用，并为我提供了类似的微调模型性能。

def dropout(input_tensor, dropout_prob):
    """Perform dropout.

    Args:
      input_tensor: float Tensor.
      dropout_prob: Python float. The probability of dropping out a value (NOT of
        *keeping* a dimension as in `tf.nn.dropout`).

    Returns:
      A version of `input_tensor` with dropout applied.
    """
    if dropout_prob is None or dropout_prob == 0.0:
        return input_tensor

    output = tf.nn.dropout(input_tensor, dropout_prob) # tf 2.10
    return output


def layer_norm(input_tensor, name=None):
    """Run layer normalization on the last dimension of the tensor."""
    input_layer_norm = tf.keras.layers.LayerNormalization(
        axis=-1, name=name, epsilon=1e-12, dtype=tf.float32)
    return input_layer_norm(input_tensor)


def layer_norm_and_dropout(input_tensor, dropout_prob, name=None):
    """Runs layer normalization followed by dropout."""
    output_tensor = layer_norm(input_tensor, name)
    # output_tensor = tf.keras.layers.Dropout(rate=dropout_prob)(output_tensor)
    output_tensor = dropout(output_tensor, dropout_prob)
    return output_tensor

需要注意的是 tf.nn.dropout需要 辍学概率而不是 保持概率与 TF1.x 版本一样，否则默认 10% 的 BERT 丢弃率会丢弃 90% 的层输出。您可以在此处引用官方变压器编码器中的更多详细信息。
https://github.com/tensorflow/models/blob/master/official/nlp/modeling/networks/transformer_encoder.py