python - BrokenProcessPool 关于在 cross_val_score 中使用 n_jobs 参数

Question

在 sklearn.model_selection.cross_val_score 中使用 n_jobs = -1 作为参数时出现错误。我是深度学习和 ANN 的初学者，根据 this 中的讲师在 k 折交叉验证中，使用 n_jobs = -1 来使用 CPU 的所有处理器以减少时间，但在我的情况下它会引发错误。

错误-

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.

可以找到完整的堆栈跟踪 here .

import keras
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score
def build_classifier():
    classifier = Sequential()
    classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu', input_dim = 11))
    classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu'))
    classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))
    classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
    return classifier
classifier = KerasClassifier(build_fn = build_classifier, batch_size = 10, nb_epoch = 100)
accuracies = cross_val_score(estimator = classifier, X = X_train, y = Y_train, cv = 10, n_jobs = -1)

Answer 1

尝试在外部文件中创建您的 build_classifier 函数并导入它。例如:

在文件 classifier_builder.py 中:

import keras

def build_classifier():
  classifier = Sequential()
  classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu', input_dim = 11))
  classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu'))
  classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))
  classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
return classifier

然后在你的笔记本上:

import classifier_builder

classifier = KerasClassifier(build_fn = build_classifier, batch_size = 10, nb_epoch = 100)
accuracies = cross_val_score(estimator = classifier, X = X_train, y = Y_train, cv = 10, n_jobs = -1)

这解决了我的问题。显然，内联函数不可 picklable。