gpt4 book ai didi

python-3.x - 如何使用word2vec进行文本分类

转载 作者:行者123 更新时间:2023-12-05 00:48:21 31 4
gpt4 key购买 nike

我想使用word2vec进行文本分类。
我得到了单词的向量。

ls = []
sentences = lines.split(".")
for i in sentences:
ls.append(i.split())
model = Word2Vec(ls, min_count=1, size = 4)
words = list(model.wv.vocab)
print(words)
vectors = []
for word in words:
vectors.append(model[word].tolist())
data = np.array(vectors)
data

输出:
array([[ 0.00933912,  0.07960335, -0.04559333,  0.10600036],
[ 0.10576613, 0.07267512, -0.10718666, -0.00804013],
[ 0.09459028, -0.09901826, -0.07074171, -0.12022413],
[-0.09893986, 0.01500741, -0.04796079, -0.04447284],
[ 0.04403428, -0.07966098, -0.06460238, -0.07369237],
[ 0.09352681, -0.03864434, -0.01743148, 0.11251986],.....])

如何进行分类(产品和非产品)?

最佳答案

您已经具有使用model.wv.syn0的单词向量数组。如果您打印它,则可以看到一个数组,其中包含单词的每个对应向量。
您可以在这里看到使用 Python3 的示例:

import pandas as pd
import os
import gensim
import nltk as nl
from sklearn.linear_model import LogisticRegression


#Reading a csv file with text data
dbFilepandas = pd.read_csv('machine learning\\Python\\dbSubset.csv').apply(lambda x: x.astype(str).str.lower())

train = []
#getting only the first 4 columns of the file
for sentences in dbFilepandas[dbFilepandas.columns[0:4]].values:
train.extend(sentences)

# Create an array of tokens using nltk
tokens = [nl.word_tokenize(sentences) for sentences in train]
现在是时候使用向量模型了,在这个例子中,我们将计算LogisticRegression。
# method 1 - using tokens in Word2Vec class itself so you don't need to train again with train method
model = gensim.models.Word2Vec(tokens, size=300, min_count=1, workers=4)

# method 2 - creating an object 'model' of Word2Vec and building vocabulary for training our model
model = gensim.models.Word2vec(size=300, min_count=1, workers=4)
# building vocabulary for training
model.build_vocab(tokens)
print("\n Training the word2vec model...\n")
# reducing the epochs will decrease the computation time
model.train(tokens, total_examples=len(tokens), epochs=4000)
# You can save your model if you want....

# The two datasets must be the same size
max_dataset_size = len(model.wv.syn0)

Y_dataset = []
# get the last number of each file. In this case is the department number
# this will be the 0 or 1, or another kind of classification. ( to use words you need to extract them differently, this way is to numbers)
with open("dbSubset.csv", "r") as f:
for line in f:
lastchar = line.strip()[-1]
if lastchar.isdigit():
result = int(lastchar)
Y_dataset.append(result)
else:
result = 40


clf = LogisticRegression(random_state=0, solver='lbfgs', multi_class='multinomial').fit(model.wv.syn0, Y_dataset[:max_dataset_size])

# Prediction of the first 15 samples of all features
predict = clf.predict(model.wv.syn0[:15, :])
# Calculating the score of the predictions
score = clf.score(model.wv.syn0, Y_dataset[:max_dataset_size])
print("\nPrediction word2vec : \n", predict)
print("Score word2vec : \n", score)
您还可以计算属于您创建的模型字典的单词的相似度:
print("\n\nSimilarity value : ",model.wv.similarity('women','men'))
您可以找到更多使用 here的函数。

关于python-3.x - 如何使用word2vec进行文本分类,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/49643974/

31 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com