c++ - C++ API 中的 Tensorflow 加载模型并出现 "from device: CUDA_ERROR_OUT_OF_MEMORY"错误

Question

我的模型大约有 2.4GB。在我的推理步骤中，我想在每个 GPU 中通过多处理方法加载模型。这意味着我尝试在一个 GPU 中做两个进程，每个进程加载一个模型。在我完成每个 session 的配置后，每个 session 获得大约5GB内存，但我仍然遇到“from device: CUDA_ERROR_OUT_OF_MEMORY”。我很纳闷。。。求助

GPU 信息:

[search@qrwt01/home/s/apps/qtfserverd/bin]$ nvidia-smi2017 年 9 月 14 日星期四 21:42:48

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.26 Driver Version: 375.26 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K80 Off | 0000:08:00.0 Off | 0 |
| N/A 48C P0 61W / 149W | 11366MiB / 11439MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K80 Off | 0000:09:00.0 Off | 0 |
| N/A 32C P0 72W / 149W | 11359MiB / 11439MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 33056 C ...ome/s/apps/qtfserverd/etc/qtfserverd.conf 5823MiB |
| 0 33057 C ...ome/s/apps/qtfserverd/etc/qtfserverd.conf 5515MiB |
| 1 33058 C ...ome/s/apps/qtfserverd/etc/qtfserverd.conf 5823MiB |
| 1 33059 C ...ome/s/apps/qtfserverd/etc/qtfserverd.conf 5516MiB |
+-----------------------------------------------------------------------------+

session 配置:

void* create_session(void* graph, std::string& checkpoint_path,
    int intra_op_threads, int inter_op_threads, std::string& device_list) {
Session* session = NULL;
SessionOptions sess_opts;
//int NUM_THREADS = 8;
if (intra_op_threads > 0) {
    sess_opts.config.set_intra_op_parallelism_threads(intra_op_threads);
}
if (inter_op_threads > 0) {
    sess_opts.config.set_inter_op_parallelism_threads(inter_op_threads);
}

sess_opts.config.set_allow_soft_placement(true);
sess_opts.config.mutable_gpu_options()->set_visible_device_list(device_list);
sess_opts.config.mutable_gpu_options()->set_allocator_type("BFC");
sess_opts.config.mutable_gpu_options()->set_per_process_gpu_memory_fraction(0.5);
sess_opts.config.mutable_gpu_options()->set_allow_growth(true);
Status status = NewSession(sess_opts, &session);
if (!status.ok()) {
    fprintf(stderr, "Create Session Failed %s\n", status.ToString().c_str());
    return NULL;
 }

错误信息

加载/home/search/tensorflow/deploy_combine.model.meta graph to/gpu:1 成功2017-09-14 21:42:31.188212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] 找到具有属性的设备 0:名称:Tesla K80 专业:3 次要:7 memoryClockRate(GHz):0.8235pciBusID:0000:09:00.0总内存:11.17GiB 空闲内存:11.05GiB2017-09-14 21:42:31.188260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1055] 创建 TensorFlow 设备 (/device:GPU:0) -> (device: 1, name: Tesla K80, pci总线 ID:0000:09:00.0，计算能力:3.7)qss_switch:1, lstm_switch:1qss_switch:1, lstm_switch:12017-09-14 21:42:33.826598: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] 无法从设备分配 1.58G(1701773312 字节):CUDA_ERROR_OUT_OF_MEMORY2017-09-14 21:42:33.838694: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] 无法从设备分配 1.43G(1531596032 字节):CUDA_ERROR_OUT_OF_MEMORY2017-09-14 21:42:33.893832: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] 无法从设备分配 439.82M(461180672 字节):CUDA_ERROR_OUT_OF_MEMORY2017-09-14 21:42:33.903917: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] 无法从设备分配 439.82M(461180672 字节):CUDA_ERROR_OUT_OF_MEMORY2017-09-14 21:42:33.913843: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] 无法从设备分配 439.82M(461180672 字节):CUDA_ERROR_OUT_OF_MEMORY2017-09-14 21:42:33.924008: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] 无法从设备分配 439.82M(461180672 字节):CUDA_ERROR_OUT_OF_MEMORY2017-09-14 21:42:33.935385: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] 无法从设备分配 439.82M(461180672 字节):CUDA_ERROR_OUT_OF_MEMORY2017-09-14 21:42:33.946556: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] 无法从设备分配 439.82M(461180672 字节):CUDA_ERROR_OUT_OF_MEMORY2017-09-14 21:42:33.956340: E tensorflow/stream_executor/cuda/cuda_driver.

Answer 1

尝试减少操作参数或分批计算，因为错误表明所有 GPU 资源已耗尽。