python - 混合数据类型的转换器

Question

我遇到了麻烦 一次将不同的变压器应用于不同类型的列 (文本与数字)，并将此类转换器连接到一个转换器中以备后用。
我尝试按照 Column Transformer with Mixed Types 的文档中的步骤进行操作，它解释了如何对分类和数字数据的混合执行此操作，但它似乎不适用于文本数据。
TL; 博士
您如何创建一个可存储的转换器，该转换器遵循不同的文本和数字数据管道？
数据下载和准备

# imports
import numpy as np

from sklearn.compose import ColumnTransformer
from sklearn.datasets import fetch_openml
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import FeatureUnion, Pipeline
from sklearn.preprocessing import StandardScaler

np.random.seed(0)

# download Titanic data
X, y = fetch_openml("titanic", version=1, as_frame=True, return_X_y=True)

# data preparation
numeric_features = ['age', 'fare']
text_features = ['name', 'cabin', 'home.dest']
X.fillna({text_col: '' for text_col in text_features}, inplace=True)

# train test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

转换数值特征:ok
按照上面链接中的步骤，可以为数字特征创建一个转换器，如下所示:

# handling missing data and normalization
numeric_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='median')),
                                      ('scaler', StandardScaler())])

num_preprocessor = ColumnTransformer(transformers=[('num', numeric_transformer, numeric_features)])

# this works
num_preprocessor.fit(X_train)
train_feature_set = num_preprocessor.transform(X_train)
test_feature_set = num_preprocessor.transform(X_test)

# verify shape = (number of data points, number of numerical features (2) )
train_feature_set.shape  # (1047, 2)
test_feature_set.shape  # (262, 2)

转换文本特征:ok
为了处理文本特征，我使用 TF-IDF 对每个文本列进行矢量化(而不是连接所有文本列，并且只应用一次 TF-IDF):

# Tfidf of max 30 features
text_transformer = TfidfVectorizer(use_idf=True,
                                   max_features=30)
# apply separately to each column
text_transformer_list = [(x + '_vectorizer', text_transformer, x) for x in text_features]
text_preprocessor = ColumnTransformer(transformers=text_transformer_list)

# this works
text_preprocessor.fit(X_train)
train_feature_set = text_preprocessor.transform(X_train)
test_feature_set = text_preprocessor.transform(X_test)

# verify shape = (number of data points, number of text features (3) times max_features(30) )
train_feature_set.shape  # (1047, 90)
test_feature_set.shape  # (262, 90)

你如何同时做到这两点？
我尝试了各种策略来将上述两个过程保存在一个变压器中，但由于不同的错误，它们都失败了。
尝试 1:遵循记录在案的策略
遵循文档( Column Transformer with Mixed Types )不起作用，一旦文本数据替换了分类数据:

# documented strategy
sum_preprocessor = ColumnTransformer(transformers=[('num', numeric_transformer, numeric_features),
                                                   ('text', text_transformer, text_features)])
# fails
sum_preprocessor.fit(X_train)

返回以下错误消息:

ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 1047 and the array at index 1 has size 3

尝试 2: FeatureUnion在变压器 list 上

# create a list of numerical transformer, like those for text
numerical_transformer_list = [(x + '_scaler', numeric_transformer, x) for x in numeric_features]

# fails
column_trans = FeatureUnion([text_transformer_list, numerical_transformer_list])

返回以下错误消息:

TypeError: All estimators should implement fit and transform. '('cabin_vectorizer', TfidfVectorizer(max_features=30), 'cabin')' (type <class 'tuple'>) doesn't

尝试 3: ColumnTransformer在变压器 list 上

# create a list of all transformers, text and numerical
sum_transformer_list = text_transformer_list + numerical_transformer_list

# works
sum_preprocessor = ColumnTransformer(transformers=sum_transformer_list)

# fails
sum_preprocessor.fit(X_train)

返回以下错误消息:

ValueError: Expected 2D array, got 1D array instead:
array=[54. nan nan ... 20. nan nan].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

我的问题
如何创建可以 fit 的单个对象和 transform混合文本和数字类型的数据？

Answer 1

简短的回答:

all_transformers = text_transformer_list + [('num', numeric_transformer, numeric_features)]

all_preprocessor = ColumnTransformer(transformers=all_transformers)

all_preprocessor.fit(X_train)
train_all = all_preprocessor.transform(X_train)
test_all = all_preprocessor.transform(X_test)

print(train_all.shape, test_all.shape)
# prints (1047, 92) (262, 92)

这里的困难在于(大多数？)文本转换器期望一维输入，但(大多数？)数值转换器期望二维输入。 ColumnTransformer通过允许您指定单列或列列表来处理这个问题:在第一种情况下，将一维数组传递给转换器，在第二种情况下传递二维数组。
因此，要解释三个尝试中的错误:
尝试 1:TF-IDF 正在接收一个二维数组，并将列视为文档而不是单个条目，因此仅产生三个输出。当它尝试将它连接到 1047 行的数字输出时，它失败了。
尝试 2: FeatureUnion没有与 ColumnTransformer 相同的输入格式:你不应该有三元组 (name, transformer, columns)在这种情况下。不管怎样， FeatureUnion不是为了你在这里做的事情。
尝试 3:这次您尝试将 1d 数据发送到数值转换器，但那些需要 2d 数据。