python - 无法重现 Xgb.cv 交叉验证结果

Question

我正在使用 Python 3.5 和 XGBoost 的 python 实现，版本 0.6

我在 Python 中构建了一个前向特征选择例程，它迭代地构建最佳特征集(导致最好的分数，这里的度量是二元分类错误)。

在我的数据集上，使用 xgb.cv 例程，通过将 max_depth(树的)增加到 40...，我可以将错误率降低到大约 0.21...

但是，如果我进行自定义交叉验证，使用相同的 XG Boost 参数、相同的折叠、相同的指标和相同的数据集，我达到的最佳分数为 0.70，max_depth 为 4 ... 如果我使用最佳通过我的 xgb.cv 例程获得的 max_depth，我的分数下降到 0.65 ...我只是不明白发生了什么...

我最好的猜测是 xgb.cv 使用不同的折叠(即在分区之前打乱数据)，但我也认为我将折叠作为输入提交给 xgb.cv(使用选项 Shuffle=False)...所以，它可能是完全不同的东西......

这是 forward_feature_selection 的代码(使用 xgb.cv):

def Forward_Feature_Selection(train, y_train, params, num_round=30, threshold=0, initial_score=0.5, to_exclude = [], nfold = 5):

    k_fold = KFold(n_splits=13)
    selected_features = []
    gain = threshold + 1
    previous_best_score = initial_score
    train = train.drop(train.columns[to_exclude], axis=1)  # df.columns is zero-based pd.Index 
    features = train.columns.values
    selected = np.zeros(len(features))
    scores = np.zeros(len(features))
    while (gain > threshold):    # we start a add-a-feature loop
        for i in range(0,len(features)):
            if (selected[i]==0):   # take only features not yet selected
                selected_features.append(features[i])
                new_train = train.iloc[:][selected_features]
                selected_features.remove(features[i])
                dtrain = xgb.DMatrix(new_train, y_train, missing = None)
            #    dtrain = xgb.DMatrix(pd.DataFrame(new_train), y_train, missing = None)
                if (i % 10 == 0):
                    print("Launching XGBoost for feature "+ str(i))
                xgb_cv = xgb.cv(params, dtrain, num_round, nfold=13, folds=k_fold, shuffle=False) 
                if params['objective'] == 'binary:logistic':
                    scores[i] = xgb_cv.tail(1)["test-error-mean"]   #classification
                else:
                    scores[i] = xgb_cv.tail(1)["test-rmse-mean"]    #regression
            else:
                scores[i] = initial_score    # discard already selected variables from candidates
        best = np.argmin(scores)
        gain = previous_best_score - scores[best]
        if (gain > 0):        
            previous_best_score = scores[best]  
            selected_features.append(features[best])
            selected[best] = 1

        print("Adding feature: " + features[best] + " increases score by " + str(gain) + ". Final score is now: " + str(previous_best_score)) 
    return (selected_features, previous_best_score)

这是我的“自定义”交叉验证:

mean_error_rate = 0
for train, test in k_fold.split(ds):
    dtrain =  xgb.DMatrix(pd.DataFrame(ds.iloc[train]), dc.iloc[train]["bin_spread"], missing = None)
    gbm = xgb.train(params, dtrain, 30)
    dtest =  xgb.DMatrix(pd.DataFrame(ds.iloc[test]), dc.iloc[test]["bin_spread"], missing = None)
    res.ix[test,"pred"] = gbm.predict(dtest)

    cv_reg = reg.fit(pd.DataFrame(ds.iloc[train]), dc.iloc[train]["bin_spread"])
    res.ix[test,"lasso"] = cv_reg.predict(pd.DataFrame(ds.iloc[test]))

    res.ix[test,"y_xgb"] = res.loc[test,"pred"] > 0.5
    res.ix[test, "xgb_right"] = (res.loc[test,"y_xgb"]==res.loc[test,"bin_spread"]) 
    print (str(100*np.sum(res.loc[test, "xgb_right"])/(N/13)))
    mean_error_rate += 100*(np.sum(res.loc[test, "xgb_right"])/(N/13))
print("mean_error_rate is : " + str(mean_error_rate/13))  

使用以下参数:

params = {"objective": "binary:logistic", 
          "booster":"gbtree",
          "max_depth":4, 
          "eval_metric" : "error",
          "eta" : 0.15}
res = pd.DataFrame(dc["bin_spread"]) 
k_fold = KFold(n_splits=13)
N = dc.shape[0]
num_trees = 30

最后调用我的前向特征选择:

selfeat = Forward_Feature_Selection(dc, 
                                    dc["bin_spread"], 
                                    params, 
                                    num_round = num_trees,
                                    threshold = 0,
                                    initial_score=999,
                                    to_exclude = [0,1,5,30,31],
                                    nfold = 13)

任何有助于理解正在发生的事情的帮助将不胜感激!提前感谢任何提示!

Answer 1

这是正常的。我也有过同样的经历。首先，Kfold 每次 split 的方式都不一样。您已在 XGBoost 中指定折叠，但 KFold split 不一致，这是正常的。接下来，模型的初始状态每次都不同。XGBoost 的内部随机状态也可能导致这种情况，请尝试更改评估指标以查看方差是否减少。如果特定指标适合您的需求，请尝试对最佳参数取平均值并将其用作您的最佳参数。