gpt4 book ai didi

python - LDA主题建模输入数据

转载 作者:太空狗 更新时间:2023-10-30 02:43:47 24 4
gpt4 key购买 nike

我是 python 新手。我刚开始从事一个项目,在推文上使用 LDA 主题建模。我正在尝试以下代码:

此示例使用在线数据集。我有一个 csv 文件,其中包含我需要使用的推文。谁能告诉我如何使用我的本地文件?我应该如何制作自己的词汇和标题?

我找不到说明如何为 LDA 准备 Material 的教程。他们都假设你已经知道如何去做。

from __future__ import division, print_function

import numpy as np
import lda
import lda.datasets


# document-term matrix

X = lda.datasets.load_reuters()
print("type(X): {}".format(type(X)))
print("shape: {}\n".format(X.shape))

# the vocab
vocab = lda.datasets.load_reuters_vocab()
print("type(vocab): {}".format(type(vocab)))
print("len(vocab): {}\n".format(len(vocab)))

# titles for each story
titles = lda.datasets.load_reuters_titles()
print("type(titles): {}".format(type(titles)))
print("len(titles): {}\n".format(len(titles)))


doc_id = 0
word_id = 3117

print("doc id: {} word id: {}".format(doc_id, word_id))
print("-- count: {}".format(X[doc_id, word_id]))
print("-- word : {}".format(vocab[word_id]))
print("-- doc : {}".format(titles[doc_id]))


model = lda.LDA(n_topics=20, n_iter=500, random_state=1)
model.fit(X)


topic_word = model.topic_word_
print("type(topic_word): {}".format(type(topic_word)))
print("shape: {}".format(topic_word.shape))


for n in range(5):
sum_pr = sum(topic_word[n,:])
print("topic: {} sum: {}".format(n, sum_pr))


n = 5
for i, topic_dist in enumerate(topic_word):
topic_words = np.array(vocab)[np.argsort(topic_dist)][:-(n+1):-1]
print('*Topic {}\n- {}'.format(i, ' '.join(topic_words)))


doc_topic = model.doc_topic_
print("type(doc_topic): {}".format(type(doc_topic)))
print("shape: {}".format(doc_topic.shape))

最佳答案

我知道这有点晚了,但希望它能有所帮助。您首先必须了解 LDA 仅适用于 DTM(文档术语矩阵)。因此,我建议您执行以下步骤:

  1. 加载您的 csv 文件
  2. 从文件中提取必要的推文
  3. 清理数据
  4. 创建一个字典,其中包含生成的语料库中的每个单词
  5. 构建 TDM 结构
  6. 使结构适合您的数据文件
  7. 获取词汇表——TDM 特征(词)
  8. 继续使用上面的代码

在这里,可以提供这段代码来帮助你入门-

token_dict = {}

for i in range(len(txt1)):
token_dict[i] = txt1[i]

len(token_dict)


print("\n Build DTM")
%time tf = CountVectorizer(stop_words='english')

print("\n Fit DTM")
%time tfs1 = tf.fit_transform(token_dict.values())

# set the number of topics to look for
num = 8

model = lda.LDA(n_topics=num, n_iter=500, random_state=1)

# we fit the DTM not the TFIDF to LDA
print("\n Fit LDA to data set")
%time model.fit_transform(tfs1)

print("\n Obtain the words with high probabilities")
%time topic_word = model.topic_word_ # model.components_ also works

print("\n Obtain the feature names")
%time vocab = tf.get_feature_names()

关于python - LDA主题建模输入数据,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/32055071/

24 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com