python - 使用 KNN 在 python 中缺失值插补

Question

我有一个看起来像这样的数据集

1908    January 5.0 -1.4
1908    February    7.3 1.9
1908    March   6.2 0.3
1908    April   NaN   2.1
1908    May NaN   7.7
1908    June    17.7    8.7
1908    July    NaN   11.0
1908    August  17.5    9.7
1908    September   16.3    8.4
1908    October 14.6    8.0
1908    November    9.6 3.4
1908    December    5.8 NaN
1909    January 5.0 0.1
1909    February    5.5 -0.3
1909    March   5.6 -0.3
1909    April   12.2    3.3
1909    May 14.7    4.8
1909    June    15.0    7.5
1909    July    17.3    10.8
1909    August  18.8    10.7  

我想使用 KNN 作为方法替换 NaN。我查找了 sklearn 的 Imputer 类，但它仅支持均值、中位数和众数插补。有一个功能请求 here但我认为到目前为止还没有实现。关于如何使用 KNN 替换最后两列中的 NaN 有什么想法吗？

编辑:由于我需要在另一个环境中运行代码，所以我没有安装包的奢侈。 Sklearn、pandas、numpy 和其他标准包是我唯一可以使用的包。

Answer 1

fancyimpute package使用以下 API 支持此类插补:

from fancyimpute import KNN    
# X is the complete data matrix
# X_incomplete has the same values as X except a subset have been replace with NaN

# Use 3 nearest rows which have a feature to fill in each row's missing features
X_filled_knn = KNN(k=3).complete(X_incomplete)

这里是这个包支持的插补:

•SimpleFill: Replaces missing entries with the mean or median of each column.

•KNN: Nearest neighbor imputations which weights samples using the mean squared difference on features for which two rows both have observed data.

•SoftImpute: Matrix completion by iterative soft thresholding of SVD decompositions. Inspired by the softImpute package for R, which is based on Spectral Regularization Algorithms for Learning Large Incomplete Matrices by Mazumder et. al.

•IterativeSVD: Matrix completion by iterative low-rank SVD decomposition. Should be similar to SVDimpute from Missing value estimation methods for DNA microarrays by Troyanskaya et. al.

•MICE: Reimplementation of Multiple Imputation by Chained Equations.

•MatrixFactorization: Direct factorization of the incomplete matrix into low-rank U and V, with an L1 sparsity penalty on the elements of U and an L2 penalty on the elements of V. Solved by gradient descent.

•NuclearNormMinimization: Simple implementation of Exact Matrix Completion via Convex Optimization by Emmanuel Candes and Benjamin Recht using cvxpy. Too slow for large matrices.

•BiScaler: Iterative estimation of row/column means and standard deviations to get doubly normalized matrix. Not guaranteed to converge but works well in practice. Taken from Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares.