keras - 如何为波斯语实现词嵌入

Question

我有此代码适用于英语但不适用于波斯语

from gensim.models import Word2Vec as wv
for sentence in sentences:
    tokens = sentence.strip().lower().split(" ")
    tokenized.append(tokens)
model = wv(tokenized
    ,size=5,
          min_count=1)
print('done2')
model.save('F:/text8/text8-phrases1')
print('done3')
print(model)
model = wv.load('F:/text8/text8-phrases1')

print(model.wv.vocab)

输出

> 'بر': <gensim.models.keyedvectors.Vocab object at 0x0000027716EEB0B8>,
> 'اساس': <gensim.models.keyedvectors.Vocab object at
> 0x0000027716EEB160>, 'قوانين': <gensim.models.keyedvectors.Vocab
> object at 0x0000027716EEB198>, 'دانشگاه':
> <gensim.models.keyedvectors.Vocab object at 0x0000027716EEB1D0>,
> 'اصفهان،': <gensim.models.keyedvectors.Vocab object at
> 0x0000027716EEB208>, 'نويسنده': <gensim.models.keyedvectors.Vocab
> object at 0x0000027716EEB240>, 'مسؤول':
> <gensim.models.keyedvectors.Vocab object at 0x0000027716EEB278>,
> 'مقاله': <gensim.models.keyedvectors.Vocab object at
> 0x0000027716EEB2B0>, 'بايد'

请以代码为例
谢谢

Answer 1

@AminST，我知道现在回答您的问题为时已晚，但可能有些人有同样的问题。所以我在这里放了一些有用的代码。我在 diikala 评论中使用了下面的代码。我只假设您有预处理部分(删除停用词、HTML、表情符号等)并且数据已准备好进行矢量化。

from hazm import word_tokenize
import pandas as pd

import gensim
from gensim.models.word2vec import Word2Vec


# reading dataset
df = pd.read_csv('data/cleaned/data.csv')
df.title = df.title.apply(str)
df.comment = df.comment.apply(str)

# Storing comments in list
comments = [comment for comment in df.comment]

# converting each sentence to list of words and inserting in sents
sents = [word_tokenize(comment) for comment in comments]

model = Word2Vec(sentences=sents, size=64, window=10, min_count=5, seed=42, workers=5)

model.save('digikala_words.w2v')

# Check for vector
model['دیجیکالا']

我真的希望它可以帮助你，我的 friend 。
如果您仍然有兴趣查看更多详细信息，请访问此链接:
digikala comment verification