tensorflow - 使用 Keras model.fit_generator 时 CUDA|RAM 内存不足

Question

我有一个形状为 (300,226,226,3) 的视频数据输入， channel 最后配置 & 我的输出是 (300,1) 存储为 numpy 数组格式。因为我不想一次加载所有数据，因为它大约有 120GB。我的代码非常简单:

import os
import sys
from random import shuffle

import numpy as np
import tensorflow as tf
from keras.layers import (BatchNormalization, Dense, Flatten, Input,
                          MaxPooling3D, TimeDistributed)
from keras.layers.convolutional import Conv3D
from keras.layers.convolutional_recurrent import ConvLSTM2D
from keras.layers.normalization import BatchNormalization
from keras.models import Model, Sequential
from keras.utils import plot_model
from model import My_ConvLSTM_Model
import numpy as np
from random import shuffle
import pandas as pd
import os


def generate_arrays(available_ids):
    datar = pd.read_csv("C:/Users/muzaf/Documents/GitHub/Data_mining/data.csv")
    while True:
        for i in available_ids:
            name_ext = str(datar.iat[i, 0])
            name = os.path.basename((os.path.splitext(name_ext))[0])
            scene = np.load('D:/Webcam/Input/{}.npy'.format(name))
            category = np.load('output/{}.npy'.format(name))
            yield (np.array([scene]), category[0])


available_ids = [i for i in range(1, 20)]

shuffle(available_ids)

final_train_id = int(len(available_ids)*0.8)
train_ids = available_ids[:final_train_id]
val_ids = available_ids[final_train_id:]
frames = 300
pixels_x = 226
pixels_y = 226
channels = 3

seq = Sequential()
seq.add(ConvLSTM2D(filters=20, kernel_size=(3, 3),
                   input_shape=(None, pixels_x, pixels_y, channels),
                   padding='same', data_format='channels_last', return_sequences=True))
seq.add(BatchNormalization())
seq.add(MaxPooling3D(pool_size=(2, 2, 1), strides=None,
                          padding='valid', data_format='channels_last'))
seq.add(TimeDistributed(Flatten()))
seq.add(TimeDistributed(Dense(32,)))
seq.add(TimeDistributed(Dense(1, activation='relu')))
seq.compile(optimizer='adam', loss='mean_squared_error', metrics=['accuracy'])
print (seq.summary())
history = seq.fit_generator(
    generate_arrays(train_ids), steps_per_epoch=len(train_ids),
    validation_data=generate_arrays(val_ids),
    validation_steps=len(val_ids),
    epochs=100, verbose=1, shuffle=False, initial_epoch=0)

一旦我运行这个，我的 GPU(GTX 1060:6GB) 内存就满了，我的 RAM 也满了。我在这里做错了什么吗？

Answer 1

尝试减少steps_per_epoch和validation_steps。把所有的数据都放在一个里会爆炸你的内存。这就像定义批量大小。

steps_per_epoch:在声明一个纪元完成并开始下一个纪元之前从生成器生成的总步骤数(样本批处理)。它通常应等于数据集的样本数除以批量大小。序列可选:如果未指定，将使用 len(generator) 作为步骤数。

validation_steps:仅当validation_data是生成器时才相关。停止之前生成器产生的总步骤数(样本批处理)。序列可选:如果未指定，将使用 len(validation_data) 作为步骤数。