python - "Resource exhausted: OOM when allocating tensor"在重新训练 GPT 2 模型 : 期间

Question

我使用 Friends Dialogues 作为数据集，使用 GPT-2 进行对话式 AI 训练，但是，它显示我内存不足。我知道这个问题已经在 StackOverflow 上得到解决，但我无法弄清楚 NLP 任务的优化。

我尝试将批量大小设置为 50(我的数据集大约有 60k 行)。我一直在关注这个tutorial在自定义数据集上重新训练 GPT-2。

我的系统规范是:
操作系统:Windows 10
内存:16GB
CPU:i7第8代
GPU:4GB 英伟达 GTX 1050Ti

这是整个错误信息

Resource exhausted: OOM when allocating tensor with shape[51200,2304] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "C:\Users\bhave\AppData\Local\conda\conda\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
    return fn(*args)
  File "C:\Users\bhave\AppData\Local\conda\conda\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Users\bhave\AppData\Local\conda\conda\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[51200,2304] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node model/h0/attn/c_attn/MatMul}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

         [[{{node Mean}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 293, in <module>
    main()
  File "train.py", line 271, in main
    feed_dict={context: sample_batch()})
  File "C:\Users\bhave\AppData\Local\conda\conda\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
    run_metadata_ptr)
  File "C:\Users\bhave\AppData\Local\conda\conda\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\bhave\AppData\Local\conda\conda\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
    run_metadata)
  File "C:\Users\bhave\AppData\Local\conda\conda\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[51200,2304] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node model/h0/attn/c_attn/MatMul (defined at D:\Python and AI\Generative Chatbot\gpt-2\src\model.py:55) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

         [[node Mean (defined at train.py:96) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


Caused by op 'model/h0/attn/c_attn/MatMul', defined at:
  File "train.py", line 293, in <module>
    main()
  File "train.py", line 93, in main
    output = model.model(hparams=hparams, X=context_in)
  File "D:\Python and AI\Generative Chatbot\gpt-2\src\model.py", line 164, in model
    h, present = block(h, 'h%d' % layer, past=past, hparams=hparams)
  File "D:\Python and AI\Generative Chatbot\gpt-2\src\model.py", line 126, in block
    a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
  File "D:\Python and AI\Generative Chatbot\gpt-2\src\model.py", line 102, in attn
    c = conv1d(x, 'c_attn', n_state*3)
  File "D:\Python and AI\Generative Chatbot\gpt-2\src\model.py", line 55, in conv1d
    c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])
  File "C:\Users\bhave\AppData\Local\conda\conda\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\math_ops.py", line 2455, in matmul
    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
  File "C:\Users\bhave\AppData\Local\conda\conda\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 5333, in mat_mul
    name=name)
  File "C:\Users\bhave\AppData\Local\conda\conda\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\bhave\AppData\Local\conda\conda\envs\tf_gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "C:\Users\bhave\AppData\Local\conda\conda\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op
    op_def=op_def)
  File "C:\Users\bhave\AppData\Local\conda\conda\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[51200,2304] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator
GPU_0_bfc
         [[node model/h0/attn/c_attn/MatMul (defined at D:\Python and AI\Generative Chatbot\gpt-2\src\model.py:55) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

         [[node Mean (defined at train.py:96) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Answer 1

我猜它总是意味着同样的事情。尝试批量大小为 1，看看是否有效。然后增加批量大小以找到您的 gpu 可以处理多少。如果它无法处理 1 的批量大小，则模型对于您的 GPU 来说可能太大了。如果您没有立即收到此错误，请检查代码是否正常，可能其中存在一些错误。哦，也许您应该检查一下还有什么正在使用您的 gpu，以确保没有任何不必要的东西占用资源。