python - 我可以向 Scikit learn Pipeline 添加异常值检测和删除吗？

Question

我想在 Scikit-Learn 中创建一个管道，其中一个特定步骤是异常值检测和删除，允许将转换后的数据传递给其他转换器和估算器。

我已搜索 SE 但无法在任何地方找到此答案。这可能吗？

Answer 1

是的。子类化 TransformerMixin 并构建自定义转换器。这是对现有异常值检测方法之一的扩展:

from sklearn.pipeline import Pipeline, TransformerMixin
from sklearn.neighbors import LocalOutlierFactor

class OutlierExtractor(TransformerMixin):
    def __init__(self, **kwargs):
        """
        Create a transformer to remove outliers. A threshold is set for selection
        criteria, and further arguments are passed to the LocalOutlierFactor class

        Keyword Args:
            neg_conf_val (float): The threshold for excluding samples with a lower
               negative outlier factor.

        Returns:
            object: to be used as a transformer method as part of Pipeline()
        """

        self.threshold = kwargs.pop('neg_conf_val', -10.0)

        self.kwargs = kwargs

    def transform(self, X, y):
        """
        Uses LocalOutlierFactor class to subselect data based on some threshold

        Returns:
            ndarray: subsampled data

        Notes:
            X should be of shape (n_samples, n_features)
        """
        X = np.asarray(X)
        y = np.asarray(y)
        lcf = LocalOutlierFactor(**self.kwargs)
        lcf.fit(X)
        return (X[lcf.negative_outlier_factor_ > self.threshold, :],
                y[lcf.negative_outlier_factor_ > self.threshold])

    def fit(self, *args, **kwargs):
        return self

然后创建一个管道:

pipe = Pipeline([('outliers', OutlierExtraction()), ...])