python - gensim 中使用 csv 的 Doc2vec

Question

我正在使用下面的乱码评论数据在 gensim 中训练 doc2vec 模型。我遇到了 2 个错误。

1st:TaggedDocument 需要 2 个参数，我无法将 Sr 字段作为第二个参数传递，因此我采用简单的字符 ('tag') 来继续下一步。

第二:当我接近 for 循环代码末尾时，出现以下错误。

ValueError:您必须指定total_examples或total_words，才能正确更新作业参数和进度计算。通常的值为total_examples=model.corpus_count。

| Sr   | review                                                     |
|------|------------------------------------------------------------|
| 123  | This is frustrating                                        |
| 456  | I am eating in a bowl and this is frustrating              |
| 678  | Summer has come and the weather is hot and I feel very hot |
| 1234 | When will winter come back I love the cool weather         |

import pandas as pd
import numpy as np
import gensim

file = pd.read_csv('/Users/test_text.csv')

file1 = [line.split() for line in file.review]

sent = [gensim.models.doc2vec.TaggedDocument(lines,'tag') for lines in file1]
model = gensim.models.Doc2Vec(alpha=0.025, min_alpha=0.025,min_count=1)  
model.build_vocab(sent)
for epoch in range(10):
        model.train(sent)
        model.alpha -= 0.002
        model.min_alpha = model.alpha 

Answer 1

我不知道如何用 Pandas 做到这一点。也就是说，使用 csv 模块，您可以执行以下操作:

import csv
from gensim.models.doc2vec import TaggedDocument, Doc2Vec 

texts = csv.DictReader(open('test_text.csv'))
documents = [TaggedDocument(text['review'].split(), [text['Sr']])  for text in texts]
model = Doc2Vec(documents, vector_size=100, window=8, min_count=2, workers=7)

# Then you can infer new vector and compute most similar documents:
vector = model.infer_vector(['frustrating', 'bowl', 'nooddle'])
print(model.docvecs.most_similar([vector]))

它会输出如下内容:

[('123', 0.07377214729785919),
 ('1234', 0.019198982045054436),
 ('456', 0.011939050629734993),
 ('678', -0.14281529188156128)]

在您的情况下，数据集适合内存，因此您无需使用开始时使用的 API。