python - sklearn.compose.ColumnTransformer() 错误地处理数组中的单个列名称？

Question

在 ColumnTransformer 中，我想使用 Tfidf 管道。但即使对于这个简单的示例，原始管道的转换输出类型和放入 ColumnTransformer 中的转换输出类型也是非常不同的类型。为什么类型不同？

多列可以作为数组 ['col1', 'col2'] 传递。为什么这不适用于单列 ['col1']？

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from nltk.stem import WordNetLemmatizer
from nltk.tokenize import word_tokenize

X = pd.DataFrame({'message': ['Lets start.', 'This is it.', 'We are done.']})

def tokenize(text):
    tokens = word_tokenize(text)
    lemmatizer = WordNetLemmatizer()

    clean_tokens = []
    for tok in tokens:
        clean_tok = lemmatizer.lemmatize(tok).lower().strip()
        clean_tokens.append(clean_tok)

    return clean_tokens

p = Pipeline(steps = [ 
            ('vect', CountVectorizer(tokenizer=tokenize)),
            ('tfidf', TfidfTransformer())
            ])  

print(p.fit_transform(X['message']))

#  (0, 6)        0.652490884512534
#  (0, 5)        0.652490884512534
#  (0, 0)        0.3853716274664007
#  (1, 7)        0.546454011634009
#  (1, 4)        0.546454011634009
#  (1, 3)        0.546454011634009
#  (1, 0)        0.3227445421804912
#  (2, 8)        0.546454011634009
#  (2, 2)        0.546454011634009
#  (2, 1)        0.546454011634009
#  (2, 0)        0.3227445421804912


ct = ColumnTransformer(transformers = [ 
        ('txt', p, ['message'])
    ])  

print(ct.fit_transform(X))

#array([[1.]])

ct = ColumnTransformer(transformers = [ 
        ('txt', p, 'message')
    ])  

print(ct.fit_transform(X))

#[[0.38537163 0.         0.         0.         0.         0.65249088
#  0.65249088 0.         0.        ]
# [0.32274454 0.         0.         0.54645401 0.54645401 0.
#  0.         0.54645401 0.        ]
# [0.32274454 0.54645401 0.54645401 0.         0.         0.
#  0.         0.         0.54645401]]

Answer 1

请参阅 documentation 中的第 6.1.4 节.

根据文档，每当转换器需要一维数组作为输入时，列就会被指定为字符串(“标题”)。对于需要 2D 数据的转换器，我们需要将列指定为字符串列表 (['title'])。