python - 未知错误/崩溃 - 带 GPU 的 TensorFlow LSTM(第一个纪元开始后无输出)

Question

我正在尝试使用 LSTM 层训练模型。我使用的是 GPU，所有需要的库都已加载。

当我以这种方式构建模型时:

model = keras.Sequential()

model.add(layers.LSTM(256, activation="relu", return_sequences=False))  # note the activation function
model.add(layers.Dropout(0.2))

model.add(layers.Dense(256, activation="relu"))
model.add(layers.Dropout(0.2))

model.add(layers.Dense(1))
model.add(layers.Activation(activation="sigmoid"))

model.compile(
    loss=keras.losses.BinaryCrossentropy(),
    optimizer="adam",
    metrics=["accuracy"]
)

它有效。但它在 LSTM 层上使用 activation="relu"，所以它不是 CuDNNLSTM - 当激活函数是 tanh(默认)时自动选择 - 如果我不是错了。

所以，它慢得令人痛苦，我想运行更快的 CuDNNLSTM。我的代码:

model = keras.Sequential()

model.add(layers.LSTM(256, return_sequences=False))
model.add(layers.Dropout(0.2))

model.add(layers.Dense(256, activation="relu"))
model.add(layers.Dropout(0.2))

model.add(layers.Dense(1))
model.add(layers.Activation(activation="sigmoid"))

model.compile(
    loss=keras.losses.BinaryCrossentropy(),
    optimizer="adam",
    metrics=["accuracy"]
)

基本一样，只是没有提供激活函数，所以会用到tanh。但是现在不是训练，输出的结尾是这样的:

2021-04-19 22:41:46.046218: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-04-19 22:41:46.046426: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-04-19 22:41:46.046642: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-04-19 22:41:46.046942: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-04-19 22:41:46.047124: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-04-19 22:41:46.047312: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-04-19 22:41:46.047489: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-04-19 22:41:46.047663: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-04-19 22:41:46.047936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-04-19 22:41:46.665456: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-19 22:41:46.665712: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2021-04-19 22:41:46.665876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2021-04-19 22:41:46.666186: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2982 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2021-04-19 22:41:46.667505: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-19 22:42:07.374456: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
Epoch 1/50
2021-04-19 22:42:08.922891: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-04-19 22:42:09.272264: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-04-19 22:42:09.302667: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll

Process finished with exit code -1073740791 (0xC0000409)

它只是开始第一个纪元，然后卡住一分钟并以这个奇怪的退出代码退出。

• 输入数据的形状:tf.Tensor([50985 29 7], shape=(3,), dtype=int32)
• 我的 GPU:Nvidia GTX 1050 Ti
• CUDA:v11.3
• 操作系统:Windows 10
• 集成开发环境:PyCharm

为这个问题寻找解决方案有点具有挑战性，因为我没有输出任何错误。难道我做错了什么？有没有人遇到过类似的问题？应该有什么帮助？

//编辑；我试过了:

• 用更少的单元(2 个而不是 256 个)和更低的 batch_size 运行这个模型
• 使用 python 3.7.1
将 tensorflow 降级到 2.4.0，将 CUDA 降级到 11.0 并将 cudnn 降级到 8.0.1(根据 this list from TensorFlow website 这应该是正确的组合)
• 重新启动我的电脑 :)

Answer 1

我找到了解决方案......有点。

因此，当我将 tensorflow 降级到 2.1.0，将 CUDA 降级到 10.1 并将 cudnn 降级到 7.6.5 时(在来自 this list on TensorFlow website 的时间第 4 个组合 )

我不知道为什么它在最新版本或 tensorflow 2.4.0 的有效组合中不起作用。

它运行良好，所以我的问题已解决。尽管如此，很高兴知道为什么在更高版本上将 LSTM 与 cudnn 结合使用对我不起作用，因为我在任何地方都没有发现这个问题。