python - 在 theano 中使用 CUDA8

Question

我安装了 CUDA8 并安装了 theano，在导入 theano 时它搜索 CUDA7.5 而不是 CUDA8，如何告诉 theano 使用 CUDA8 而不是 CUDA7.5？

我的系统只有 CUDA8，不包含混合环境 cuda(即同时具有 CUDA7.5 和 CUDA8)。

这是 nvidia-smi 的输出

$ nvidia-smi 
Sat Feb  4 11:32:30 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.26                 Driver Version: 375.26                         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 970M    Off  | 0000:01:00.0     Off |                  N/A |
| N/A   54C    P0    22W /  N/A |      0MiB /  3016MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |

这是 nvcc -V

的输出

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Sun_Sep__4_22:14:01_CDT_2016
Cuda compilation tools, release 8.0, V8.0.44

在 ipython 中导入 theano 时，它无法在 gpu 模式下运行，错误是找不到 libcudart.so.7.5

Python 3.6.0 (default, Jan 16 2017, 12:12:55) 
Type "copyright", "credits" or "license" for more information.

IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import theano
ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: libcudart.so.7.5: cannot open shared object file: No such file or directory
WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu0 is not available  (error: cuda unavailable)

这是我的.theanorc

的内容

[global]                                                                                                                                                                                                       
floatX = float32
device = gpu0
cuda.root = /opt/cuda

我尝试从源代码构建 theano，在卸载它之前的安装后，它也不起作用。我确实使用 theano-cache clean/theano-cache purge 清除了 theano-cache 并手动删除了 .theano 目录下的内容，这也帮不上忙。

随着更多的调试，我在这里得到错误 https://github.com/Theano/Theano/blob/8b9f73365e4932f1c005a0a37b907d28985fbc5f/theano/gof/cmodule.py#L302

当nvcc_compiler尝试从theano缓存中的cuda_ndarray加载cuda_ndarray.so

mod.cu 的编译阶段运行无误。

在这种情况下，链接器指向错误的 libcudart

readelf -a cuda_ndarray.so | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libcublas.so.8.0]
 0x0000000000000001 (NEEDED)             Shared library: [libpython3.6m.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libcudart.so.7.5]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

我假设 ldconfig 正确缓存了 cuda 库

$ sudo ldconfig -v | grep -e 'cuda\|blas'
/opt/cuda/lib64:
    libcublas.so.8.0 -> libcublas.so.8.0.45
    libcudart.so.8.0 -> libcudart.so.8.0.44
    libnvblas.so.8.0 -> libnvblas.so.8.0.44
/opt/cuda/nvvm/lib64:
    libcuda.so.1 -> libcuda.so.375.26
    libblas.so.3 -> libblas.so.3.7.0
    libicudata.so.58 -> libicudata.so.58.2
    libopenblas.so.0 -> libopenblas.so
    libicudata.so.58 -> libicudata.so.58.1

Answer 1

在深入研究我的问题后，我重构了我原来的问题并张贴在这里 nvcc is picking wrong libcudart library这解决了我的问题。