python - 如何解决 google colab 中的 ResourceExhaustedError

Question

我在 google colab 上使用 roberta 来解决推文情绪提取问题的问答模型。

但模型无法训练，因为我收到 Resourceexhaustederror；

查看完整错误:

ResourceExhaustedError:  OOM when allocating tensor with shape[32,16,128,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[node model/tf_roberta_model/roberta/encoder/layer_._17/attention/self/transpose (defined at /usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_tf_roberta.py:218) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
 [Op:__inference_train_function_112984]...

在这里查看模型:

ids = Input((MAX_LEN,), dtype=tf.int32)
att = Input((MAX_LEN,), dtype=tf.int32)

bert_model = TFRobertaModel.from_pretrained('roberta-large')

x = bert_model(ids, attention_mask= att)

x1 = Dropout(0.1)(x[0])
x1 = Conv1D(1,1)(x1)
x1 = Flatten()(x1)
x1 = Activation('softmax')(x1)


x2 = Dropout(0.1)(x[0])
x2 = Conv1D(1,1)(x2)
x2 = Flatten()(x2)
x2 = Activation('softmax')(x2)


model = Model(inputs = [ids, att], outputs = [x1, x2])

如能帮助解决此错误，我们将不胜感激。

Answer 1

根据我的经验，您可以使用 Gradient Accumulation 技术。或者，更好的选择，如果您可以设法使用 Google Colab Pro .根据文档

With Colab Pro you get priority access to high-memory VMs. These VMs generally have double the memory of standard Colab VMs, and twice as many CPUs. You will be able to access a notebook setting to enable high-memory VMs once you are subscribed. Additionally, you may sometimes be automatically assigned a high-memory VM when Colab detects that you are likely to need it. Colab Pro VMs also generally come with double the disk of standard Colab VMs. Resources are not guaranteed, though, and there are usage limits for high memory VMs.

In the free version of Colab, the high-memory preference is not available, and users are rarely automatically assigned high memory VMs.

这些transformer模型很吃内存，用Colab Pro很方便。不过，您也可以使用 Colab 中也提供的 TPU 加速器，但请注意，它比 Kaggle TPU 慢得多。