azure - 简单的 TensorFlow 计算无法在不同系统(macOS、Colab、Azure)上重现

Question

我正在研究 TensorFlow 中代码在我的 macOS 计算机、Google Colab 以及使用 Docker 的 Azure 上的再现性。我知道我可以设置图形级种子和操作级种子。我使用的是 eager 模式(因此没有并行优化)并且没有 GPU。我使用从单位法线中随机抽取的 100x100 数据并计算它们的平均值和标准差。

下面的测试代码验证我没有使用 GPU，我使用的是 Tensorflow 1.12.0 或 TensorFlow 2 预览版，张量 if Float32 、随机张量的第一个元素(如果我仅设置图级种子或操作级种子，则其具有不同的值)、它们的平均值和标准差。我还设置了 NumPy 的随机种子，尽管我在这里没有使用它:

import numpy as np
import tensorflow as tf


def tf_1():
    """Returns True if TensorFlow is version 1"""
    return tf.__version__.startswith("1.")


def format_number(n):
    """Returns the number string-formatted with 12 number after comma."""
    return "%1.12f" % n


def set_top_level_seeds():
    """Sets TensorFlow graph-level seed and Numpy seed."""
    if tf_1():
        tf.set_random_seed(0)
    else:
        tf.random.set_seed(0)
    np.random.seed(0)

def generate_random_numbers(op_seed=None):
    """Returns random normal draws"""

    if op_seed:
        t = tf.random.normal([100, 100], seed=op_seed)
    else:
        t = tf.random.normal([100, 100])  

    return t    

def generate_random_number_stats_str(op_seed=None):
    """Returns mean and standard deviation from random normal draws"""

    t = generate_random_numbers(op_seed = op_seed)

    mean = tf.reduce_mean(t)
    sdev = tf.sqrt(tf.reduce_mean(tf.square(t - mean)))

    return [format_number(n) for n in (mean, sdev)]


def generate_random_number_1_seed():
    """Returns a single random number with graph-level seed only."""
    set_top_level_seeds()
    num = generate_random_numbers()[0, 0]
    return num


def generate_random_number_2_seeds():
    """Returns a single random number with graph-level seed only."""
    set_top_level_seeds()
    num = generate_random_numbers(op_seed=1)[0, 0]
    return num


def generate_stats_1_seed():
    """Returns mean and standard deviation wtih graph-level seed only."""
    set_top_level_seeds()
    return generate_random_number_stats_str()


def generate_stats_2_seeds():
    """Returns mean and standard deviation with graph and operation seeds."""
    set_top_level_seeds()
    return generate_random_number_stats_str(op_seed=1)


class Tests(tf.test.TestCase):
    """Run tests for reproducibility of TensorFlow."""

    def test_gpu(self):
        self.assertEqual(False, tf.test.is_gpu_available())

    def test_version(self):
        self.assertTrue(tf.__version__ == "1.12.0" or
                        tf.__version__.startswith("2.0.0-dev2019"))

    def test_type(self):
        num_type = generate_random_number_1_seed().dtype
        self.assertEqual(num_type, tf.float32)

    def test_eager_execution(self):
        self.assertEqual(True, tf.executing_eagerly())

    def test_random_number_1_seed(self):
        num_str = format_number(generate_random_number_1_seed())
        self.assertEqual(num_str, "1.511062622070")

    def test_random_number_2_seeds(self):
        num_str = format_number(generate_random_number_2_seeds())
        self.assertEqual(num_str, "0.680345416069")

    def test_arithmetic_1_seed(self):
        m, s = generate_stats_1_seed()

        if tf_1():
            self.assertEqual(m, "-0.008264393546")
            self.assertEqual(s, "0.995371103287")
        else:
            self.assertEqual(m, "-0.008264398202")
            self.assertEqual(s, "0.995371103287")

    def test_arithmetic_2_seeds(self):
        m, s = generate_stats_2_seeds()

        if tf_1():
            self.assertEqual(m, "0.000620653736")
            self.assertEqual(s, "0.997191190720")
        else:
            self.assertEqual(m, "0.000620646286")
            self.assertEqual(s, "0.997191071510")


if __name__ == '__main__':

    tf.reset_default_graph()

    if tf_1():
        tf.enable_eager_execution()
        tf.logging.set_verbosity(tf.logging.ERROR)

    tf.test.main()

在我的本地计算机上，在我使用 pip install tensorflow==1.12.0 安装 Tensorflow 的虚拟环境中使用 TensorFlow 1.12.0 或 TensorFlow 2 预览版进行了测试。或pip install tf-nightly-2.0-preview 。请注意，两个版本中的第一次随机抽取是相同的，因此我假设所有随机数都是相同的，但平均值和标准差在小数点后 9 位之后是不同的。因此TensorFlow在不同版本中实现计算的方式略有不同。

在 Google Colab 上，我将最后一个命令替换为 import unittest; unittest.main(argv=['first-arg-is-ignored'], exit=False) (参见this issue)。所有测试都禁止一次通过:相同的随机数、相同的均值和标准差以及图形级种子。失败的测试是图级种子和操作级种子的平均值算术，差异从小数点后第九位开始:

.F.......
======================================================================
FAIL: test_arithmetic_2_seeds (__main__.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython-input-7-16d0afebf95f>", line 109, in test_arithmetic_2_seeds
    self.assertEqual(m, "0.000620653736")
AssertionError: '0.000620654086' != '0.000620653736'
- 0.000620654086
?           ^^^
+ 0.000620653736
?           ^^^


----------------------------------------------------------------------
Ran 9 tests in 0.023s

FAILED (failures=1)

在 Azure 上使用 Standard_NV6机器与 NVIDIA GPU Cloud Image ，以及以下 Dockerfile

FROM tensorflow/tensorflow:latest-py3
ADD tests.py .
CMD python tests.py

在仅图级种子和图级和操作级种子的两种情况下，算术测试均失败:

FF.......
======================================================================
FAIL: test_arithmetic_1_seed (__main__.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 99, in test_arithmetic_1_seed
    self.assertEqual(m, "-0.008264393546")
AssertionError: '-0.008264395408' != '-0.008264393546'
- -0.008264395408
?              ^^
+ -0.008264393546
?            +  ^


======================================================================
FAIL: test_arithmetic_2_seeds (__main__.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 109, in test_arithmetic_2_seeds
    self.assertEqual(m, "0.000620653736")
AssertionError: '0.000620655250' != '0.000620653736'
- 0.000620655250
+ 0.000620653736


----------------------------------------------------------------------
Ran 9 tests in 0.016s

FAILED (failures=2)

当测试在 Google Colab 或 Azure 上失败时，它们的失败结果与平均值的实际值一致，因此我相信问题不在于我可以设置的其他随机种子。

为了查看问题是否是 TensorFlow 在不同系统上的实现，我在 Azure 上使用 TensorFlow 的不同图像( tensorflow/tensorflow:latest ，没有 -py3 标签)以及具有顶级种子的随机数进行测试也不同:

FF..F....
======================================================================
FAIL: test_arithmetic_1_seed (__main__.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 99, in test_arithmetic_1_seed
    self.assertEqual(m, "-0.008264393546")
AssertionError: '0.001101632486' != '-0.008264393546'

======================================================================
FAIL: test_arithmetic_2_seeds (__main__.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 109, in test_arithmetic_2_seeds
    self.assertEqual(m, "0.000620653736")
AssertionError: '0.000620655250' != '0.000620653736'

======================================================================
FAIL: test_random_number_1_seed (__main__.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 89, in test_random_number_1_seed
    self.assertEqual(num_str, "1.511062622070")
AssertionError: '-1.398459434509' != '1.511062622070'

----------------------------------------------------------------------
Ran 9 tests in 0.015s

如何确保 TensorFlow 计算在不同系统上的再现性？

Answer 1

浮点计算的精度取决于库编译选项和系统架构细节。

有很多文章讨论了可靠地比较 float 是否相等的困难。搜索“浮点相等”就会找到它们。一个例子是 https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/