python - 检查 scikitlearn imputer 丢弃了哪些特征

Question

docs对于 scikit-learn 的 Imputation transformer 说

When axis=0, columns which only contained missing values at fit are discarded upon transform.

由于 imputer 返回一个 numpy 数组，我如何检查哪些特征在插补过程中被丢弃，或者相应地，哪些特征在插补后被保留？

这是一个简单的例子:

import pandas as pd
import numpy as np
from sklearn.preprocessing import Imputer

df = pd.DataFrame(np.random.randn(10, 5), columns=['a', 'b', 'c', 'd', 'e'])
df['f'] = len(df3)*['NaN']

这是数据框:

>>> df
      a         b         c         d         e    f
0 -1.284658  0.246541 -1.120987  0.559911 -1.189870  NaN
1  0.773717  0.430597 -0.004346 -1.292080  1.993266  NaN
2  1.418761 -0.004749 -0.181932 -0.305756 -0.135870  NaN
3  0.418673 -0.376318 -0.860783  0.074135 -1.034095  NaN
4 -0.019873  0.006210  0.364384  1.029895 -0.188727  NaN
5  0.903661  0.123575 -0.556970  1.344985 -1.109806  NaN
6 -0.069168 -0.385597  0.684345  0.645920  1.159898  NaN
7  0.695782  0.030239 -0.777304 -0.037102  2.053028  NaN
8 -0.256409  0.106735 -0.729710  0.254626  1.064925  NaN
9  0.235507 -0.087767  0.626121  1.391286  0.449158  NaN

现在我创建一个 imputer imp:

imp = Imputer(missing_values='NaN', strategy='mean', axis=0)
imp.fit(df)
imputed = imp.transform(df)

这是从插补返回的 numpy 数组。

>>> imputed
array([[-1.28465763,  0.24654083, -1.12098675,  0.55991059, -1.18986998],
   [ 0.77371694,  0.43059674, -0.0043461 , -1.29208032,  1.99326594],
   [ 1.41876145, -0.0047488 , -0.18193164, -0.30575631, -0.13586974],
   [ 0.41867326, -0.37631792, -0.86078293,  0.07413458, -1.03409532],

Answer 1

如何检查插补过程中丢弃了哪些特征？

包含所有 NaN 的列将被丢弃。您可以在不通过 df.isnull().all() 的 fit 和 transform 过程的情况下进行检查。在 True 的地方，那些是将被丢弃的“特征”。

不过，确切的答案是像这样将 verbose=1 添加到您的 imputer 中:

imp = Imputer(verbose=1)

为了让这个例子更清楚发生了什么，向 df 添加另一列包含所有 NaN。

df.insert(2, 'g', np.nan)

df 现在看起来像这样:

          a         b   g         c         d         e   f
0 -1.284658  0.246541 NaN -1.120987  0.559911 -1.189870 NaN
1  0.773717  0.430597 NaN -0.004346 -1.292080  1.993266 NaN
2  1.418761 -0.004749 NaN -0.181932 -0.305756 -0.135870 NaN
3  0.418673 -0.376318 NaN -0.860783  0.074135 -1.034095 NaN
4 -0.019873  0.006210 NaN  0.364384  1.029895 -0.188727 NaN
5  0.903661  0.123575 NaN -0.556970  1.344985 -1.109806 NaN
6 -0.069168 -0.385597 NaN  0.684345  0.645920  1.159898 NaN
7  0.695782  0.030239 NaN -0.777304 -0.037102  2.053028 NaN
8 -0.256409  0.106735 NaN -0.729710  0.254626  1.064925 NaN
9  0.235507 -0.087767 NaN  0.626121  1.391286  0.449158 NaN

正在运行...

imp.fit(df)
imp.transform(df)

现在输出以下“详细”消息，告诉您删除了哪些列 [2 6]:

Warning (from warnings module):
  File "C:\Python34\lib\site-packages\sklearn\preprocessing\imputation.py", line 347
    "observed values: %s" % missing)
UserWarning: Deleting features without observed values: [2 6]
array([[-1.284658,  0.246541, -1.120987,  0.559911, -1.18987 ],
       [ 0.773717,  0.430597, -0.004346, -1.29208 ,  1.993266],
       [ 1.418761, -0.004749, -0.181932, -0.305756, -0.13587 ],
       [ 0.418673, -0.376318, -0.860783,  0.074135, -1.034095],
       [-0.019873,  0.00621 ,  0.364384,  1.029895, -0.188727],
       [ 0.903661,  0.123575, -0.55697 ,  1.344985, -1.109806],
       [-0.069168, -0.385597,  0.684345,  0.64592 ,  1.159898],
       [ 0.695782,  0.030239, -0.777304, -0.037102,  2.053028],
       [-0.256409,  0.106735, -0.72971 ,  0.254626,  1.064925],
       [ 0.235507, -0.087767,  0.626121,  1.391286,  0.449158]])

插补后保留了哪些特征？

插补后保留的列和值。

使用我之前的 df，如果我们添加一些 NaN 到组合中:

df.loc[[1, 7, 3], ['a', 'c', 'e']] = np.nan

df 看起来像这样:

          a         b   g         c         d         e   f
0 -1.284658  0.246541 NaN -1.120987  0.559911 -1.189870 NaN
1       NaN  0.430597 NaN       NaN -1.292080       NaN NaN
2  1.418761 -0.004749 NaN -0.181932 -0.305756 -0.135870 NaN
3       NaN -0.376318 NaN       NaN  0.074135       NaN NaN
4 -0.019873  0.006210 NaN  0.364384  1.029895 -0.188727 NaN
5  0.903661  0.123575 NaN -0.556970  1.344985 -1.109806 NaN
6 -0.069168 -0.385597 NaN  0.684345  0.645920  1.159898 NaN
7       NaN  0.030239 NaN       NaN -0.037102       NaN NaN
8 -0.256409  0.106735 NaN -0.729710  0.254626  1.064925 NaN
9  0.235507 -0.087767 NaN  0.626121  1.391286  0.449158 NaN

重要的是要了解您使用的是什么插补策略。 Imputer 的默认值是mean。这意味着它将用给定列的平均值替换 NaN 值。

为了证明这一点，首先检查每列的平均值:

>>> df.mean()
a    0.132546
b    0.008947
g         NaN
c   -0.130678
d    0.366582
e    0.007101
f         NaN
dtype: float64

然后您可以进行拟合和转换，并查看转换后的估算数据中是否有任何值在 imp.statistics_ 超参数中。

imp = Imputer(verbose=1)
imp.fit(df)
imp.transform(df)

返回以下内容 - 同样，需要注意的关键是 NaN 值已替换为给定列的 mean。例如，无论您在第一列中看到 0.13254586，您都会注意到它们出现在第 1、3 和 7 行(以前是 NaN):

Warning (from warnings module):
  File "C:\Python34\lib\site-packages\sklearn\preprocessing\imputation.py", line 347
    "observed values: %s" % missing)
UserWarning: Deleting features without observed values: [2 6]
array([[-1.284658  ,  0.246541  , -1.120987  ,  0.559911  , -1.18987   ],
       [ 0.13254586,  0.430597  , -0.13067843, -1.29208   ,  0.00710114],
       [ 1.418761  , -0.004749  , -0.181932  , -0.305756  , -0.13587   ],
       [ 0.13254586, -0.376318  , -0.13067843,  0.074135  ,  0.00710114],
       [-0.019873  ,  0.00621   ,  0.364384  ,  1.029895  , -0.188727  ],
       [ 0.903661  ,  0.123575  , -0.55697   ,  1.344985  , -1.109806  ],
       [-0.069168  , -0.385597  ,  0.684345  ,  0.64592   ,  1.159898  ],
       [ 0.13254586,  0.030239  , -0.13067843, -0.037102  ,  0.00710114],
       [-0.256409  ,  0.106735  , -0.72971   ,  0.254626  ,  1.064925  ],
       [ 0.235507  , -0.087767  ,  0.626121  ,  1.391286  ,  0.449158  ]])

如果您想进行 bool 比较以查看估算了哪些值，您可以执行以下操作(不是万无一失，但最可靠的方法):

np.reshape(np.in1d(imp.transform(df), imp.statistics_), imp.transform(df).shape)
array([[False, False, False, False, False],
       [ True, False,  True, False,  True],
       [False, False, False, False, False],
       [ True, False,  True, False,  True],
       [False, False, False, False, False],
       [False, False, False, False, False],
       [False, False, False, False, False],
       [ True, False,  True, False,  True],
       [False, False, False, False, False],
       [False, False, False, False, False]], dtype=bool)