gpt4 book ai didi

scala - Spark MLib Word2Vec 错误 : The vocabulary size should be > 0

转载 作者:行者123 更新时间:2023-12-01 19:05:44 29 4
gpt4 key购买 nike

我正在尝试使用 Spark 的 MLLib 实现词向量化。我按照给出的例子 here .

我有一堆句子,我想将它们作为输入来训练模型。但我不确定这个模型是否采用句子或仅将所有单词作为字符串序列。

我的输入如下:

scala> v.take(5)
res31: Array[Seq[String]] = Array(List([WrappedArray(0_42)]), List([WrappedArray(big, baller, shoe, ?)]), List([WrappedArray(since, eliud, win, ,, quick, fact, from, runner, from, country, kalenjins, !, write, ., happy, quick, fact, kalenjins, location, :, kenya, (, kenya's, western, highland, rift, valley, ), population, :, 4, ., 9, million, ;, compose, 11, subtribes, language, :, kalenjin, ;, swahili, ;, english, church, :, christianity, ~, africa, inland, church, [, aic, ],, church, province, kenya, [, cpk, ],, roman, catholic, church, ;, islam, translation, :, kalenjin, translate, ", tell, ", formation, :, wwii, ,, gikuyu, tribal, member, wish, separate, create, identity, ., later, ,, student, attend, alliance, high, school, (, first, british, public, school, kenya, ), form, ...

但是当我尝试在此输入上训练我的 word2vec 模型时,它不起作用。

scala> val word2vec = new Word2Vec()
word2vec: org.apache.spark.mllib.feature.Word2Vec = org.apache.spark.mllib.feature.Word2Vec@51567040

scala> val model = word2vec.fit(v)
java.lang.IllegalArgumentException: requirement failed: The vocabulary size should be > 0. You may need to check the setting of minCount, which could be large enough to remove all your words in sentences.

Word2Vec 不接受句子作为输入吗?

最佳答案

您的输入正确。但是,Word2Vec 将自动删除词汇表中出现次数未达到最低次数的单词(所有句子组合)。默认情况下,该值为 5。在您的情况下,很可能没有任何单词在您使用的数据中出现 5 次或以上。

要更改所需的最少单词出现次数,请使用 setMinCount(),例如最小计数为 2:

val word2vec = new Word2Vec().setMinCount(2)

关于scala - Spark MLib Word2Vec 错误 : The vocabulary size should be > 0,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/59557524/

29 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com