python - Keras Sklearn RandomizedSearchCV GPU OOM 错误

Question

我想将 RandomizedSearchCV(或 GridSearchCV)应用于我的 Keras 模型(TensorFlow 后端)。然而，在使用不同的超参数集训练几次后，出现了OOM错误。

下面是我的代码和错误信息。我该如何解决这个问题？提前致谢。

def build_model(num_filters = 10, 
                num_classes = 6,
                sequence_max_length=512, 
                vocab_size=71, 
                embedding_size=16, 
                learning_rate=0.001, 
                dropout = 0.2,
                top_k=3,
                embedding_matrix = None,
                model_path=None):
    ... do something
    return model

keras_model = KerasClassifier(build_fn=build_model, 
    embedding_matrix = embedding_matrix) 

random_search_model = RandomizedSearchCV(keras_model, 
                        n_iter = 5,
                        param_distributions = hparm_dist,
                        refit = True,
                        n_jobs = 1)

错误信息:

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[471512,300]

更新:

通过将 keras.backend.clear_session() 添加到 sklearn.cross_validation._fit_and_score 的末尾来解决。

Answer 1

您可以尝试将“pre_dispatch”参数更改为 1。By default it is 2*n_jobs

random_search_model = RandomizedSearchCV(keras_model, 
                    n_iter = 5,
                    param_distributions = hparm_dist,
                    refit = True,
                    pre_dispatch=1,
                    n_jobs = 1)