machine-learning - Keras运行时错误: GpuCorrMM failed to allocate working memory of 576 x 802816

Question

我尝试在 Keras 中运行深度学习代码，但始终收到以下错误消息。我四处寻找并花了很多时间但仍然无法修复它。我是一条鱼，任何帮助将不胜感激!!

runfile('E:/dilation-keras/predict.py', wdir='E:/dilation-keras')
Using Theano backend.
Using gpu device 0: GeForce GT 635M (CNMeM is enabled with initial size: 90.0% of memory, cuDNN not available)
____________________________________________________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to                     
====================================================================================================
input_3 (InputLayer)             (None, 3, 900, 900)   0                                            
____________________________________________________________________________________________________
conv1_1 (Convolution2D)          (None, 64, 898, 898)  1792        input_3[0][0]                    
____________________________________________________________________________________________________
conv1_2 (Convolution2D)          (None, 64, 896, 896)  36928       conv1_1[0][0]                    
____________________________________________________________________________________________________
pool1 (MaxPooling2D)             (None, 64, 448, 448)  0           conv1_2[0][0]                    
____________________________________________________________________________________________________
conv2_1 (Convolution2D)          (None, 128, 446, 446) 73856       pool1[0][0]                      
____________________________________________________________________________________________________
conv2_2 (Convolution2D)          (None, 128, 444, 444) 147584      conv2_1[0][0]                    
____________________________________________________________________________________________________
pool2 (MaxPooling2D)             (None, 128, 222, 222) 0           conv2_2[0][0]                    
____________________________________________________________________________________________________
conv3_1 (Convolution2D)          (None, 256, 220, 220) 295168      pool2[0][0]                      
____________________________________________________________________________________________________
conv3_2 (Convolution2D)          (None, 256, 218, 218) 590080      conv3_1[0][0]                    
____________________________________________________________________________________________________
conv3_3 (Convolution2D)          (None, 256, 216, 216) 590080      conv3_2[0][0]                    
____________________________________________________________________________________________________
pool3 (MaxPooling2D)             (None, 256, 108, 108) 0           conv3_3[0][0]                    
____________________________________________________________________________________________________
conv4_1 (Convolution2D)          (None, 512, 106, 106) 1180160     pool3[0][0]                      
____________________________________________________________________________________________________
conv4_2 (Convolution2D)          (None, 512, 104, 104) 2359808     conv4_1[0][0]                    
____________________________________________________________________________________________________
conv4_3 (Convolution2D)          (None, 512, 102, 102) 2359808     conv4_2[0][0]                    
____________________________________________________________________________________________________
conv5_1 (AtrousConvolution2D)    (None, 512, 98, 98)   2359808     conv4_3[0][0]                    
____________________________________________________________________________________________________
conv5_2 (AtrousConvolution2D)    (None, 512, 94, 94)   2359808     conv5_1[0][0]                    
____________________________________________________________________________________________________
conv5_3 (AtrousConvolution2D)    (None, 512, 90, 90)   2359808     conv5_2[0][0]                    
____________________________________________________________________________________________________
fc6 (AtrousConvolution2D)        (None, 4096, 66, 66)  102764544   conv5_3[0][0]                    
____________________________________________________________________________________________________
drop6 (Dropout)                  (None, 4096, 66, 66)  0           fc6[0][0]                        
____________________________________________________________________________________________________
fc7 (Convolution2D)              (None, 4096, 66, 66)  16781312    drop6[0][0]                      
____________________________________________________________________________________________________
drop7 (Dropout)                  (None, 4096, 66, 66)  0           fc7[0][0]                        
____________________________________________________________________________________________________
fc-final (Convolution2D)         (None, 21, 66, 66)    86037       drop7[0][0]                      
____________________________________________________________________________________________________
zeropadding2d_3 (ZeroPadding2D)  (None, 21, 132, 132)  0           fc-final[0][0]                   
____________________________________________________________________________________________________
ct_conv1_1 (Convolution2D)       (None, 42, 130, 130)  7980        zeropadding2d_3[0][0]            
____________________________________________________________________________________________________
ct_conv1_2 (Convolution2D)       (None, 42, 128, 128)  15918       ct_conv1_1[0][0]                 
____________________________________________________________________________________________________
ct_conv2_1 (AtrousConvolution2D) (None, 84, 124, 124)  31836       ct_conv1_2[0][0]                 
____________________________________________________________________________________________________
ct_conv3_1 (AtrousConvolution2D) (None, 168, 116, 116) 127176      ct_conv2_1[0][0]                 
____________________________________________________________________________________________________
ct_conv4_1 (AtrousConvolution2D) (None, 336, 100, 100) 508368      ct_conv3_1[0][0]                 
____________________________________________________________________________________________________
ct_conv5_1 (AtrousConvolution2D) (None, 672, 68, 68)   2032800     ct_conv4_1[0][0]                 
____________________________________________________________________________________________________
ct_fc1 (Convolution2D)           (None, 672, 66, 66)   4064928     ct_conv5_1[0][0]                 
____________________________________________________________________________________________________
ct_final (Convolution2D)         (None, 21, 66, 66)    14133       ct_fc1[0][0]                     
____________________________________________________________________________________________________
permute_5 (Permute)              (None, 66, 66, 21)    0           ct_final[0][0]                   
____________________________________________________________________________________________________
reshape_5 (Reshape)              (None, 4356, 21)      0           permute_5[0][0]                  
____________________________________________________________________________________________________
activation_3 (Activation)        (None, 4356, 21)      0           reshape_5[0][0]                  
____________________________________________________________________________________________________
reshape_6 (Reshape)              (None, 66, 66, 21)    0           activation_3[0][0]               
____________________________________________________________________________________________________
permute_6 (Permute)              (None, 21, 66, 66)    0           reshape_6[0][0]                  
====================================================================================================
Total params: 141,149,720
Trainable params: 141,149,720
Non-trainable params: 0
____________________________________________________________________________________________________
batch_size is: 1
Traceback (most recent call last):

  File "<ipython-input-3-641fac717a39>", line 1, in <module>
    runfile('E:/dilation-keras/predict.py', wdir='E:/dilation-keras')

  File "c:\users\lenovo\anaconda2\lib\site-packages\spyder\utils\site\sitecustomize.py", line 866, in runfile
    execfile(filename, namespace)

  File "c:\users\lenovo\anaconda2\lib\site-packages\spyder\utils\site\sitecustomize.py", line 87, in execfile
    exec(compile(scripttext, filename, 'exec'), glob, loc)

  File "E:/dilation-keras/predict.py", line 74, in <module>
    y_img = predict(im, model, ds)

  File "E:/dilation-keras/predict.py", line 46, in predict
    prob = model.predict(model_in,batch_size=batch_size)[0]

  File "c:\users\lenovo\anaconda2\lib\site-packages\keras\engine\training.py", line 1272, in predict
    batch_size=batch_size, verbose=verbose)

  File "c:\users\lenovo\anaconda2\lib\site-packages\keras\engine\training.py", line 945, in _predict_loop
    batch_outs = f(ins_batch)

  File "c:\users\lenovo\anaconda2\lib\site-packages\keras\backend\theano_backend.py", line 959, in __call__
    return self.function(*inputs)

  File "c:\users\lenovo\anaconda2\lib\site-packages\theano\compile\function_module.py", line 886, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))

  File "c:\users\lenovo\anaconda2\lib\site-packages\theano\gof\link.py", line 325, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)

  File "c:\users\lenovo\anaconda2\lib\site-packages\theano\compile\function_module.py", line 873, in __call__
    self.fn() if output_subset is None else\

RuntimeError: GpuCorrMM failed to allocate working memory of 576 x 802816

Apply node that caused the error: GpuCorrMM{valid, (1, 1), (1, 1)}(GpuContiguous.0, GpuContiguous.0)
Toposort index: 95
Inputs types: [CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, 4D)]
Inputs shapes: [(1, 64, 898, 898), (64, 64, 3, 3)]
Inputs strides: [(0, 806404, 898, 1), (576, 9, 3, 1)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[GpuElemwise{Composite{(i0 * ((i1 + i2) + Abs((i1 + i2))))}}[(0, 1)](CudaNdarrayConstant{[[[[ 0.5]]]]}, GpuCorrMM{valid, (1, 1), (1, 1)}.0, GpuDimShuffle{x,0,x,x}.0)]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

Answer 1

问题出在内存上。您的前十个 conv2D 层每层大约需要 200MB。这意味着仅存储前 10 层的输出就需要 2GB。这肯定不适合你的卡内存。