performance - Tensorflow GPU 利用率仅为 60% (GTX 1070)

Question

我正在使用 tensorflow 训练 CNN 模型。我的 GPU 利用率仅达到 60% (+- 2-3%)，没有大幅下降。

Sun Oct 23 11:34:26 2016       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.57                 Driver Version: 367.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070    Off  | 0000:01:00.0     Off |                  N/A |
|  1%   53C    P2    90W / 170W |   7823MiB /  8113MiB |     60%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      3644    C   /usr/bin/python2.7                            7821MiB |
+-----------------------------------------------------------------------------+

因为它是 Pascal 卡，所以我使用 CUDA 8 和 cudnn 5.1.5CPU使用率约为50%(均匀分布在8个线程上。i7 4770k)，因此CPU性能不应成为瓶颈。

我使用 Tensorflow 的二进制文件格式并使用 tf.TFRecordReader() 读取

我正在创建像这样的批量图像:

#Uses tf.TFRecordReader() to read single Example
label, image = read_and_decode_single_example(filename_queue=filename_queue) 
image = tf.image.decode_jpeg(image.values[0], channels=3)
jpeg = tf.cast(image, tf.float32) / 255.
jpeg.set_shape([66,200,3])
images_batch, labels_batch = tf.train.shuffle_batch(
    [jpeg, label], batch_size= FLAGS.batch_size,
    num_threads=8,
    capacity=2000, #tried bigger values here, does not change the performance
    min_after_dequeue=1000) #here too

这是我的训练循环:

sess = tf.Session()

sess.run(init)
tf.train.start_queue_runners(sess=sess)
for step in xrange(FLAGS.max_steps):
    labels, images = sess.run([labels_batch, images_batch])
    feed_dict = {images_placeholder: images, labels_placeholder: labels}
    _, loss_value = sess.run([train_op, loss],
                                 feed_dict=feed_dict)

我对 tensorflow 没有太多经验，而且我现在不知道瓶颈可能在哪里。如果您需要更多代码片段来帮助识别问题，我将提供它们。

更新:带宽测试结果

==5172== NVPROF is profiling process 5172, command: ./bandwidthtest

Device: GeForce GTX 1070
Transfer size (MB): 3960

Pageable transfers
  Host to Device bandwidth (GB/s): 7.066359
  Device to Host bandwidth (GB/s): 6.850315

Pinned transfers
  Host to Device bandwidth (GB/s): 12.038037
  Device to Host bandwidth (GB/s): 12.683915

==5172== Profiling application: ./bandwidthtest
==5172== Profiling result:
Time(%)      Time     Calls       Avg       Min       Max  Name
 50.03%  933.34ms         2  466.67ms  327.33ms  606.01ms  [CUDA memcpy DtoH]
 49.97%  932.32ms         2  466.16ms  344.89ms  587.42ms  [CUDA memcpy HtoD]

==5172== API calls:
Time(%)      Time     Calls       Avg       Min       Max  Name
 46.60%  1.86597s         4  466.49ms  327.36ms  606.15ms  cudaMemcpy
 35.43%  1.41863s         2  709.31ms  632.94ms  785.69ms  cudaMallocHost
 17.89%  716.33ms         2  358.17ms  346.14ms  370.19ms  cudaFreeHost
  0.04%  1.5572ms         1  1.5572ms  1.5572ms  1.5572ms  cudaMalloc
  0.02%  708.41us         1  708.41us  708.41us  708.41us  cudaFree
  0.01%  203.58us         1  203.58us  203.58us  203.58us  cudaGetDeviceProperties
  0.00%  187.55us         1  187.55us  187.55us  187.55us  cuDeviceTotalMem
  0.00%  162.41us        91  1.7840us     105ns  61.874us  cuDeviceGetAttribute
  0.00%  79.979us         4  19.994us  1.9580us  73.537us  cudaEventSynchronize
  0.00%  77.074us         8  9.6340us  1.5860us  28.925us  cudaEventRecord
  0.00%  19.282us         1  19.282us  19.282us  19.282us  cuDeviceGetName
  0.00%  17.891us         4  4.4720us     629ns  8.6080us  cudaEventDestroy
  0.00%  16.348us         4  4.0870us     818ns  8.8600us  cudaEventCreate
  0.00%  7.3070us         4  1.8260us  1.7040us  2.0680us  cudaEventElapsedTime
  0.00%  1.6670us         3     555ns     128ns  1.2720us  cuDeviceGetCount
  0.00%     813ns         3     271ns     142ns     439ns  cuDeviceGet

Answer 1

在获得更多 tensorflow 经验后，我意识到 GPU 的使用很大程度上取决于网络大小、批量大小和预处理。使用具有更多转换层的更大网络(例如 Resnet 样式)会增加 GPU 使用率，因为涉及更多计算，并且通过传输数据等产生的开销(与计算相关)更少。