machine-learning - 管道中 CountVectorizer 的 Sklearn NotFittedError

Question

我正在尝试学习如何通过 sklearn 处理文本数据，但遇到了一个无法解决的问题。

我正在关注的教程是:http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html

输入是具有两列的 pandas df。一种带有文本，一种带有二进制类。

代码:

from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

traindf, testdf = train_test_split(nlp_df, stratify=nlp_df['class'])

x_train = traindf['text']
x_test = traindf['text']
y_train = traindf['class']
y_test = testdf['class']

# CV
count_vect = CountVectorizer(stop_words='english')
x_train_modified = count_vect.fit_transform(x_train)
x_test_modified = count_vect.transform(x_test)


# TF-IDF
idf = TfidfTransformer()
fit = idf.fit(x_train_modified)
x_train_mod2 = fit.transform(x_train_modified)

# MNB

mnb = MultinomialNB()
x_train_data = mnb.fit(x_train_mod2, y_train)

text_clf = Pipeline([('vect', CountVectorizer()),
             ('tfidf', TfidfTransformer()),
               ('clf', MultinomialNB()),
                ])

predicted = text_clf.predict(x_test_modified)

当我尝试运行最后一行时:

---------------------------------------------------------------------------
NotFittedError                            Traceback (most recent call last)
<ipython-input-64-8815003b4713> in <module>()
----> 1 predicted = text_clf.predict(x_test_modified)

~/anaconda3/lib/python3.6/site-packages/sklearn/utils/metaestimators.py in <lambda>(*args, **kwargs)
    113 
    114         # lambda, but not partial, allows help() to work with update_wrapper
--> 115         out = lambda *args, **kwargs: self.fn(obj, *args, **kwargs)
    116         # update the docstring of the returned function
    117         update_wrapper(out, self.fn)

~/anaconda3/lib/python3.6/site-packages/sklearn/pipeline.py in predict(self, X)
    304         for name, transform in self.steps[:-1]:
    305             if transform is not None:
--> 306                 Xt = transform.transform(Xt)
    307         return self.steps[-1][-1].predict(Xt)
    308 

~/anaconda3/lib/python3.6/site-packages/sklearn/feature_extraction/text.py in transform(self, raw_documents)
    918             self._validate_vocabulary()
    919 
--> 920         self._check_vocabulary()
    921 
    922         # use the same matrix-building strategy as fit_transform

~/anaconda3/lib/python3.6/site-packages/sklearn/feature_extraction/text.py in _check_vocabulary(self)
    301         """Check if vocabulary is empty or missing (not fit-ed)"""
    302         msg = "%(name)s - Vocabulary wasn't fitted."
--> 303         check_is_fitted(self, 'vocabulary_', msg=msg),
    304 
    305         if len(self.vocabulary_) == 0:

~/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py in check_is_fitted(estimator, attributes, msg, all_or_any)
    766 
    767     if not all_or_any([hasattr(estimator, attr) for attr in attributes]):
--> 768         raise NotFittedError(msg % {'name': type(estimator).__name__})
    769 
    770 

NotFittedError: CountVectorizer - Vocabulary wasn't fitted.

关于如何修复此错误有什么建议吗？我正在根据测试数据正确转换 CV 模型。我什至检查了词汇列表是否为空，它不是 (count_vect.vocabulary_)

谢谢!

Answer 1

您的问题有几个问题。

对于初学者来说，您实际上并不适合管道，因此会出现错误。更仔细地查看 linked tutorial ，您会看到有一个步骤 text_clf.fit (其中 text_clf 确实是管道)。

其次，您没有正确使用管道的概念，它恰好适合端到端的整个内容；相反，您将其各个组件一一适配...如果您再次查看教程，您将看到管道适配的代码:

text_clf.fit(twenty_train.data, twenty_train.target)  

使用初始形式的数据，而不像您一样使用中间转换；本教程的重点是演示如何将各个转换封装在管道中(并替换为管道)，而不是在这些转换之上使用管道...

第三，您应该避免将变量命名为 fit - 这是一个保留关键字；同样，我们不使用 CV 来缩写 Count Vectorizer(在 ML 术语中，CV 代表交叉验证)。

也就是说，这是使用管道的正确方法:

from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

traindf, testdf = train_test_split(nlp_df, stratify=nlp_df['class'])

x_train = traindf['text']
x_test = traindf['text']
y_train = traindf['class']
y_test = testdf['class']

text_clf = Pipeline([('vect', CountVectorizer(stop_words='english')),
                    ('tfidf', TfidfTransformer()),
                    ('clf', MultinomialNB()),
                     ])

text_clf.fit(x_train, y_train) 

predicted = text_clf.predict(x_test)

正如您所看到的，管道的目的是让事情变得更简单(与逐个顺序使用组件相比)，而不是让它们进一步复杂化......