python - Windows 10、RTX 2070] : Failed to get convolution algorithm

Question

我目前正在尝试使用去年发布的深度学习算法(https://github.com/talmo/leap)来分析小鼠的行为。到目前为止，我已经使用了 Quadro P400，它与 CUDA 9.0 配合得很好。然而，我购买了 RTX 2070，因为我需要更多的计算能力。由于RTX卡仅与CUDA 10.0兼容，我尝试执行新安装(计算机与以前不一样，它是全新的)，但我已经遇到这个问题好几天了，到目前为止我还不能找到解决方法。我尝试了不同的解决方案，如这里提到的 https://github.com/tensorflow/tensorflow/issues/24828 。我还尝试按照 https://www.pytorials.com/how-to-install-tensorflow-gpu-with-cuda-10-0-for-python-on-windows/ 来编译自己的tensorflow它有效，但我在尝试运行该算法时遇到了相同的错误。

系统信息

OS Platform and Distribution : Windows 10 Pro
TensorFlow installed from (source or binary): Source and Binary (tried both)
TensorFlow version: 1.12
Python version: 3.6.6
Installed using virtualenv? pip? conda?: pip and conda (tried both)
Bazel version (if compiling from source): 0.16.1
CUDA/cuDNN version: Cudnn - 7.4.2 , CUDA- 10.0
GPU model and memory: GeForce RTX 2070

我尝试过不同版本的 Cdunn，基本上是 cuda 10.0 的所有版本、其他版本的 Python(3.7.1、3.6.4)和 Tensorflow(1.13.1，每晚构建)。

我不知道下一步可以尝试什么，所以我请求您的帮助。

提供您在遇到问题之前执行的命令/步骤的确切顺序

任何其他信息/日志

总参数:592,066可训练参数:592,066不可训练参数:0

<小时/>

Created folder: C:\Users\dieudon\Downloads\models\190512_222333-n=17 
Epoch 1/15 
Traceback (most recent call last): 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call 
    return fn(*args) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn 
    options, feed_dict, fetch_list, target_list, run_metadata) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun 
    run_metadata) 
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. 
     [[{{node conv2d_1/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:@training/Adam/gradients/conv2d_1/convolution_grad/Conv2DBackpropFilter"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/gradients/conv2d_1/convolution_grad/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]] 
     [[{{node loss/mul/_287}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1575_loss/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] 
 
During handling of the above exception, another exception occurred: 
 
Traceback (most recent call last): 
  File "C:\Users\dieudon\Desktop\Matlab\leap-master\leap\training.py", line 276, in <module> 
    clize.run(train) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\sigtools\modifiers.py", line 158, in __call__ 
    return self.func(*args, **kwargs) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\clize\runner.py", line 360, in run 
    ret = cli(*args) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\clize\runner.py", line 220, in __call__ 
    return func(*posargs, **kwargs) 
  File "C:\Users\dieudon\Desktop\Matlab\leap-master\leap\training.py", line 255, in train 
    viz_grid_callback 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper 
    return func(*args, **kwargs) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\keras\engine\training.py", line 2230, in fit_generator 
    class_weight=class_weight) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\keras\engine\training.py", line 1883, in train_on_batch 
    outputs = self.train_function(ins) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py", line 2482, in __call__ 
    **self.session_kwargs) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 929, in run 
    run_metadata_ptr) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run 
    feed_dict_tensor, options, run_metadata) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run 
    run_metadata) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call 
    raise type(e)(node_def, op, message) 
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. 
     [[node conv2d_1/convolution (defined at C:\Users\dieudon\Anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py:3341)  = Conv2D[T=DT_FLOAT, _class=["loc:@training/Adam/gradients/conv2d_1/convolution_grad/Conv2DBackpropFilter"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/gradients/conv2d_1/convolution_grad/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]] 
     [[{{node loss/mul/_287}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1575_loss/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] 
 
Caused by op 'conv2d_1/convolution', defined at: 
  File "C:\Users\dieudon\Desktop\Matlab\leap-master\leap\training.py", line 276, in <module> 
    clize.run(train) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\sigtools\modifiers.py", line 158, in __call__ 
    return self.func(*args, **kwargs) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\clize\runner.py", line 360, in run 
    ret = cli(*args) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\clize\runner.py", line 220, in __call__ 
    return func(*posargs, **kwargs) 
  File "C:\Users\dieudon\Desktop\Matlab\leap-master\leap\training.py", line 191, in train 
    model = create_model(net_name, img_size, num_output_channels, filters=filters, amsgrad=amsgrad, upsampling_layers=upsampling_layers, summary=True) 
  File "C:\Users\dieudon\Desktop\Matlab\leap-master\leap\training.py", line 104, in create_model 
    return compile_model(img_size, output_channels, **kwargs) 
  File "c:\users\dieudon\desktop\matlab\leap-master\leap\models.py", line 23, in leap_cnn 
    x1 = Conv2D(filters, kernel_size=3, padding="same", activation="relu")(x_in) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\keras\engine\topology.py", line 619, in __call__ 
    output = self.call(inputs, **kwargs) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\keras\layers\convolutional.py", line 168, in call 
    dilation_rate=self.dilation_rate) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py", line 3341, in conv2d 
    data_format=tf_data_format) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 780, in convolution 
    return op(input, filter) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 868, in __call__ 
    return self.conv_op(inp, filter) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 520, in __call__ 
    return self.call(inp, filter) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 204, in __call__ 
    name=self.name) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 1044, in conv2d 
    data_format=data_format, dilations=dilations, name=name) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper 
    op_def=op_def) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func 
    return func(*args, **kwargs) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3274, in create_op 
    op_def=op_def) 
  File "C:\Users\dieudon\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1770, in __init__ 
    self._traceback = tf_stack.extract_stack() 
 
UnknownError (see above for traceback): Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. 
     [[node conv2d_1/convolution (defined at C:\Users\dieudon\Anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py:3341)  = Conv2D[T=DT_FLOAT, _class=["loc:@training/Adam/gradients/conv2d_1/convolution_grad/Conv2DBackpropFilter"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/gradients/conv2d_1/convolution_grad/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]] 
     [[{{node loss/mul/_287}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1575_loss/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] 

如何解决这个问题？

Answer 1

您所要做的就是在代码开头添加以下几行:

from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession

config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)