python - 相同形式的数据集有两种不同的形状

Question

我对机器学习还很陌生，刚刚掌握了这些技术。因此，我尝试使用具有 4 个特征和目标特征/类(真值 1 或 0)的数据集在以下分类器上训练模型。

分类器

• SGD 分类器
• 随机森林分类器
• 线性支持向量分类器
• 高斯过程分类器

我正在以下数据集上训练模型[部分数据集如下所示]。

训练集:train_sop_truth.csv

Subject,Predicate,Object,Computed,Truth
concept:sportsteam:hawks,concept:teamplaysincity,concept:city:atlanta,0.4255912602,1
concept:stadiumoreventvenue:honda+AF8-center,concept:stadiumlocatedincity,concept:city:anaheim,0.4276425838,1
concept:sportsteam:ducks,concept:teamplaysincity,concept:city:anaheim,0.4762486517,1
concept:sportsteam:n1985+AF8-chicago+AF8-bears,concept:teamplaysincity,concept:city:chicago,0.4106097221,1
concept:stadiumoreventvenue:philips+AF8-arena,concept:stadiumlocatedincity,concept:city:atlanta,0.4190083146,1
concept:stadiumoreventvenue:united+AF8-center,concept:stadiumlocatedincity,concept:city:chicago,0.4211134315,1

测试数据集位于另一个 .csv 文件中，名称为 test_sop_truth.csv。

测试集:test_sop_truth.csv

Subject,Predicate,Object,Computed,Truth
Nigel_Cole,isMarriedTo,Kate_Isitt,0.9350595474,1
Véra_Clouzot,isMarriedTo,Henri-Georges_Clouzot,0.4773990512,1
Norodom_Sihanouk,produced,The_Last_Days_of_Colonel_Savath,0.3942225575,1
Farouk_of_Egypt,isMarriedTo,Farida_of_Egypt,0.4276426733,1

然后我想检查每个数据集的特征形状，并期望看到与我对两个数据集应用相同转换时相同数量的特征。但他们的看法不同。

Python代码

import pandas as pd
import numpy as np
from termcolor import colored

features = pd.read_csv('../Data/train_sop_truth.csv')
testFeatures = pd.read_csv('../Data/test_sop_truth.csv')
print(features.head(5))

print(colored('\nThe shape of our features is:','green'), features.shape)
print(colored('\nThe shape of our Test features is:','green'), testFeatures.shape)

print()
print(colored('\n     DESCRIPTIVE STATISTICS\n','yellow'))
print(colored(features.describe(),'cyan'))
print()
print(colored(testFeatures.describe(),'cyan'))


features = pd.get_dummies(features)
testFeatures = pd.get_dummies(testFeatures)

features.iloc[:,5:].head(5)
testFeatures.iloc[:,5].head(5)

labels = np.array(features['Truth'])
testlabels = np.array(testFeatures['Truth'])


features= features.drop('Truth', axis = 1)
testFeatures = testFeatures.drop('Truth', axis = 1)

feature_list = list(features.columns)
testFeature_list = list(testFeatures.columns)

features = np.array(features)
testFeatures = np.array(testFeatures)

train_samples = 100


testX_train, textX_test, testy_train, testy_test = model_selection.train_test_split(testFeatures, testlabels, test_size=0.25, random_state = 42)

X_train, X_test, y_train, y_test = model_selection.train_test_split(features, labels, test_size = 0.25, random_state = 42)

print(colored('\n    TRAINING & TESTING SETS','yellow'))
print(colored('\nTraining Features Shape:','magenta'), X_train.shape)
print(colored('Training Labels Shape:','magenta'), X_test.shape)
print(colored('Testing Features Shape:','magenta'), y_train.shape)
print(colored('Testing Labels Shape:','magenta'), y_test.shape)

print()

print(colored('\n    TRAINING & TESTING SETS','yellow'))
print(colored('\nTraining Features Shape:','magenta'), testX_train.shape)
print(colored('Training Labels Shape:','magenta'), textX_test.shape)
print(colored('Testing Features Shape:','magenta'), testy_train.shape)
print(colored('Testing Labels Shape:','magenta'), testy_test.shape)

输出

The shape of our features is: (1860, 5)

The shape of our Test features is: (1386, 5)


     DESCRIPTIVE STATISTICS

          Computed        Truth
count  1860.000000  1860.000000
mean      0.443222     0.913441
std       0.110788     0.281264
min       0.000000     0.000000
25%       0.418164     1.000000
50%       0.427643     1.000000
75%       0.450023     1.000000
max       1.000000     1.000000

          Computed        Truth
count  1386.000000  1386.000000
mean      0.511809     0.992063
std       0.197954     0.088765
min       0.009042     0.000000
25%       0.418649     1.000000
50%       0.429140     1.000000
75%       0.515809     1.000000
max       1.702856     1.000000

    TRAINING & TESTING SETS

Training Features Shape: (1395, 1045)
Training Labels Shape: (465, 1045)
Testing Features Shape: (1395,)
Testing Labels Shape: (465,)


    TRAINING & TESTING SETS

Training Features Shape: (1039, 1790)
Training Labels Shape: (347, 1790)
Testing Features Shape: (1039,)
Testing Labels Shape: (347,)

我在这里不明白的是特征形状如何不同，如特征(训练集)的 1045 和 testFeatures(测试集)的 1790，尽管 csv 文件中经历了相同的转换并且具有相同数量的特征和特征形式。

在这方面的任何建议或澄清将不胜感激。

Answer 1

当您对测试数据集应用 get_dummies 时，您可能会根据分类变量的数据值添加或删除几列。

def add_missing_dummy_columns( d, columns ):
        missing_cols = set( columns ) - set( d.columns )
        for c in missing_cols:
            d[c] = 0

def fix_columns( d, columns ):  

    add_missing_dummy_columns( d, columns )

    # make sure we have all the columns we need
    assert( set( columns ) - set( d.columns ) == set())

    extra_cols = set( d.columns ) - set( columns )
    if extra_cols: print ("extra columns:", extra_cols)

    d = d[ columns ]
    return d

testFeatures= fix_columns( testFeatures, features.columns )