machine-learning - Word2Vec - 具有高交叉验证分数的模型对测试数据的表现非常糟糕

Question

在对 twitter 数据进行情感分析时，我遇到了一个我无法解决的问题。我想训练一个随机森林分类器来检测仇恨言论。因此，我使用了一个带有标记的数据集，其中的推文被标记为 1 表示仇恨言论，0 表示正常推文。对于矢量化，我使用 Word2Vec。我首先进行了一个超参数化来为分类器找到好的参数。
在超参数化期间，我使用了重复的分层 KFold 交叉验证(评分 = 准确性)
这里的平均准确率约为 99.6%。然而，一旦我将模型应用于测试数据集并绘制混淆矩阵，准确率仅高于 50%，这对于二元分类器来说当然是糟糕的。
我成功地对 Bag of Words 使用了完全相同的方法，并且在这里完全没有问题。
有人可以快速查看我的代码吗？那会很有帮助。我就是找不到哪里出了问题。非常感谢!
(我还将代码上传到 google collab，以防您更容易:https://colab.research.google.com/drive/15BzElijL3vwa_6DnLicxRvcs4SPDZbpe?usp=sharing)
首先我预处​​理我的数据:

train_csv = pd.read_csv(r'/content/drive/My Drive/Colab Notebooks/MLDA_project/data2/train.csv')
train = train_csv     
#check for missing values (result shows that there are no missing values)
train.isna().sum()    
# remove the tweet IDs
train.drop(train.columns[0], axis = "columns", inplace = True)    
# create a new column to save the cleansed tweets
train['training_tweet'] = np.nan

# remove special/unknown characters
train.replace('[^a-zA-Z#]', ' ', inplace = True, regex = True)    
# generate stopword list and add the twitter handles "user" to the stopword list
stopwords = sw.words('english')
stopwords.append('user')    
# convert to lowercase
train = train.applymap(lambda i:i.lower() if type(i) == str else i)    
# execute tokenization and lemmatization
lemmatizer = WordNetLemmatizer()

for i in range(len(train.index)):
    #tokenize the tweets from the column "tweet"
    words = nltk.word_tokenize(train.iloc[i, 1])
    #consider words with more than 3 characters
    words = [word for word in words if len(word) > 3] 
    #exclude words in stopword list
    words = [lemmatizer.lemmatize(word) for word in words if word not in set(stopwords)] 
    #Join words again
    train.iloc[i, 2]  = ' '.join(words)  
    words = nltk.word_tokenize(train.iloc[i, 2])
train.drop(train.columns[1], axis = "columns", inplace = True)

majority = train[train.label == 0]
minority = train[train.label == 1]
# upsample minority class
minority_upsampled = resample(minority, replace = True, n_samples = len(majority))      
# combine majority class with upsampled minority class
train_upsampled = pd.concat([majority, minority_upsampled])
train = train_upsampled
np.random.seed(10)
train = train.sample(frac = 1)
train = train.reset_index(drop = True)

现在 train有第 0 列的标签和第 1 列的预处理推文。
接下来我定义了 Word2Vec Vectorizer:

def W2Vvectorize(X_train):
tokenize=X_train.apply(lambda x: x.split())
w2vec_model=gensim.models.Word2Vec(tokenize,min_count = 1, size = 100, window = 5, sg = 1)
w2vec_model.train(tokenize,total_examples= len(X_train), epochs=20)
w2v_words = list(w2vec_model.wv.vocab)
vector=[]
from tqdm import tqdm
for sent in tqdm(tokenize):
    sent_vec=np.zeros(100)
    count =0
    for word in sent: 
        if word in w2v_words:
            vec = w2vec_model.wv[word]
            sent_vec += vec 
            count += 1
    if count != 0:
        sent_vec /= count #normalize
    vector.append(sent_vec)
return vector

我将数据集拆分为测试集和训练集，并使用上述定义的 W2V 对两个子集进行矢量化:

x = train["training_tweet"]
y = train["label"]

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, stratify=train['label'])

print('X Train Shape = total * 0,8 =', X_train.shape)
print('y Train Shape = total * 0,8 =', y_train.shape)
print('X Test Shape = total * 0,2 =', X_test.shape)
print('y Test Shape = total * 0,2 =', y_test.shape) # change 0,4 & 0,6

train_tf_w2v = W2Vvectorize(X_train)
test_tf_w2v = W2Vvectorize(X_test)

现在我进行超参数化:

# define models and parameters
model = RandomForestClassifier()
n_estimators = [10, 100, 1000]
max_features = ['sqrt', 'log2']
# define grid search
grid = dict(n_estimators=n_estimators,max_features=max_features)
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
grid_search = GridSearchCV(estimator=model, param_grid=grid, n_jobs=-1, cv=cv, scoring='accuracy',error_score=0)
grid_result = grid_search.fit(train_tf_w2v, y_train)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))

这导致以下输出:

Best: 0.996628 using {'max_features': 'log2', 'n_estimators': 1000}
0.995261 (0.000990) with: {'max_features': 'sqrt', 'n_estimators': 10}
0.996110 (0.000754) with: {'max_features': 'sqrt', 'n_estimators': 100}
0.996081 (0.000853) with: {'max_features': 'sqrt', 'n_estimators': 1000}
0.995885 (0.000872) with: {'max_features': 'log2', 'n_estimators': 10}
0.996481 (0.000691) with: {'max_features': 'log2', 'n_estimators': 100}
0.996628 (0.000782) with: {'max_features': 'log2', 'n_estimators': 1000}

接下来，我想使用模型绘制带有测试数据的混淆矩阵:

clf = RandomForestClassifier(max_features = 'log2', n_estimators=1000) 
   
clf.fit(train_tf_w2v, y_train)
name = clf.__class__.__name__
        
expectation = y_test
test_prediction = clf.predict(test_tf_w2v)
acc = accuracy_score(expectation, test_prediction)   
pre = precision_score(expectation, test_prediction)
rec = recall_score(expectation, test_prediction)
f1 = f1_score(expectation, test_prediction)

fig, ax = plt.subplots(1,2, figsize=(14,4))
plt.suptitle(f'{name} \n', fontsize = 18)
plt.subplots_adjust(top = 0.8)
skplt.metrics.plot_confusion_matrix(expectation, test_prediction, ax=ax[0])
skplt.metrics.plot_confusion_matrix(expectation, test_prediction, normalize=True, ax = ax[1])
plt.show()
    
print(f"for the {name} we receive the following values:")
print("Accuracy: {:.3%}".format(acc))
print('Precision score: {:.3%}'.format(pre))
print('Recall score: {:.3%}'.format(rec))
print('F1 score: {:.3%}'.format(f1))

这输出:

对于 RandomForestClassifier，我们收到以下值:
准确率:57.974%
精度得分:99.790%
召回分数:15.983%
F1 分数:27.552%

Answer 1

呃...现在我觉得自己很愚蠢。我发现出了什么问题。
在训练/测试分割之后，我将两个子集独立发送到 W2Vvectorize()功能。

train_tf_w2v = W2Vvectorize(X_train)
test_tf_w2v = W2Vvectorize(X_test)

从那里 W2Vvectorize()函数基于两个独立的子集训练两个独立的 Word2Vec 模型。因此，当我通过矢量化测试数据时 test_tf_w2v对于我训练有素的 RandomForest 分类器，为了检查测试集的准确性是否也正确，在训练有素的 RandomForest 分类器看来，就好像测试集将使用不同的语言一样。两个独立的 word2vec 模型只是以不同的方式进行矢量化。
我解决了如下:

def W2Vvectorize(X_train):
    tokenize=X_train.apply(lambda x: x.split())
    vector=[]
    for sent in tqdm(tokenize):
        sent_vec=np.zeros(100)
        count =0
        for word in sent: 
            if word in w2v_words:
                vec = w2vec_model.wv[word]
                sent_vec += vec 
                count += 1
        if count != 0:
            sent_vec /= count #normalize
        vector.append(sent_vec)
    return vector

Word2Vec 训练与此分开:

x = train["training_tweet"]
y = train["label"]

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, stratify=train['label'])

print('X Train Shape = total * 0,8 =', X_train.shape)
print('y Train Shape = total * 0,8 =', y_train.shape)
print('X Test Shape = total * 0,2 =', X_test.shape)
print('y Test Shape = total * 0,2 =', y_test.shape) #

tokenize=X_train.apply(lambda x: x.split())
w2vec_model=gensim.models.Word2Vec(tokenize,min_count = 1, size = 100, window = 5, sg = 1)
w2vec_model.train(tokenize,total_examples= len(X_train), epochs=20)
w2v_words = list(w2vec_model.wv.vocab)

train_tf_w2v = W2Vvectorize(X_train)
test_tf_w2v = W2Vvectorize(X_test)

所以 Word2Vec 模型训练只在训练数据上进行。然而，测试数据的矢量化必须使用完全相同的 Word2Vec 模型进行。