Python:如何将字数列表转换为适合 CountVectorizer 的格式

Question

我有大约 100,000 个以下形式的字符串列表:
['the: 652', 'of: 216', 'in: 168', 'to: 159', 'is: 145'] 等
这基本上构成了我的语料库。每个列表包含文档中的单词及其字数。

如何将此语料库放入可以输入 CountVectorizer 的表单中？

是否有比将每个列表转换为包含 652 次“the”、216 次“of”等的字符串更快的方法？

Answer 1

假设您想要实现的是稀疏矩阵格式的矢量化语料库以及经过训练的矢量化器，您可以在不重复数据的情况下模拟矢量化过程:

from scipy.sparse.lil import lil_matrix
from sklearn.feature_extraction.text import CountVectorizer

corpus = [['the: 652', 'of: 216', 'in: 168', 'to: 159', 'is: 145'],
          ['king: 20', 'of: 16', 'the: 400', 'jungle: 110']]


# Prepare a vocabulary for the vectorizer
vocabulary = {item.split(':')[0] for document in corpus for item in document}
indexed_vocabulary = {term: index for index, term in enumerate(vocabulary)}
vectorizer = CountVectorizer(vocabulary=indexed_vocabulary)

# Vectorize the corpus using the coordinates known to the vectorizer
X = lil_matrix((len(corpus), len(vocabulary)))
X.data = [[int(item.split(':')[1]) for item in document] for document in corpus]
X.rows = [[vectorizer.vocabulary[(item.split(':')[0])] for item in document]
          for document in corpus]

# Convert the matrix to csr format to be compatible with vectorizer.transform output
X = X.tocsr()

在此示例中，输出将是:

[[ 168.  216.    0.  159.  652.  145.    0.]
 [   0.   16.  110.    0.  400.    0.   20.]]

这可以允许进一步的文档矢量化:

vectorizer.transform(['jungle kid is programming', 'the jungle machine learning jungle'])

其产量:

[[0 0 1 0 0 1 0]
 [0 0 2 0 1 0 0]]