gpt4 book ai didi

java - Spark : StringIndexer on sentences

转载 作者:行者123 更新时间:2023-11-30 02:18:05 26 4
gpt4 key购买 nike

我正在尝试对一列句子执行 StringIndexer 操作,即将单词列表转换为整数列表。

例如:

输入数据集:

  (1, ["I", "like", "Spark"])
(2, ["I", "hate", "Spark"])

我预计 StringIndexer 之后的输出如下:

  (1, [0, 2, 1])
(2, [0, 3, 1])

理想情况下,我希望将此类转换作为管道的一部分进行,以便我可以将变压器链接在一起并序列化以进行在线服务。

这是 Spark 本身支持的吗?

谢谢!

最佳答案

用于将文本转换为特征的标准TransformersCountVectorizer

CountVectorizer and CountVectorizerModel aim to help convert a collection of text documents to vectors of token counts.

HashingTF :

Maps a sequence of terms to their term frequencies using the hashing trick. Currently we use Austin Appleby's MurmurHash 3 algorithm (MurmurHash3_x86_32) to calculate the hash code value for the term object. Since a simple modulo is used to transform the hash function to a column index, it is advisable to use a power of two as the numFeatures parameter; otherwise the features will not be mapped evenly to the columns.

两者都有binary选项,可用于从计数 vector 切换到二进制 vector 。

没有内置的 Transfomer 可以给出您想要的精确结果(它对 ML 算法没有用),您可以explode 应用 StringIndexercollect_list/collect_set:

import org.apache.spark.ml.feature._
import org.apache.spark.ml.Pipeline


val df = Seq(
(1, Array("I", "like", "Spark")), (2, Array("I", "hate", "Spark"))
).toDF("id", "words")

val pipeline = new Pipeline().setStages(Array(
new SQLTransformer()
.setStatement("SELECT id, explode(words) as word FROM __THIS__"),
new StringIndexer().setInputCol("word").setOutputCol("index"),
new SQLTransformer()
.setStatement("""SELECT id, COLLECT_SET(index) AS values
FROM __THIS__ GROUP BY id""")
))

pipeline.fit(df).transform(df).show

// +---+---------------+
// | id| values|
// +---+---------------+
// | 1|[0.0, 1.0, 3.0]|
// | 2|[2.0, 0.0, 1.0]|
// +---+---------------+

使用CountVectorizerudf:

import org.apache.spark.ml.linalg._


spark.udf.register("indices", (v: Vector) => v.toSparse.indices)

val pipeline = new Pipeline().setStages(Array(
new CountVectorizer().setInputCol("words").setOutputCol("vector"),
new SQLTransformer()
.setStatement("SELECT *, indices(vector) FROM __THIS__")
))

pipeline.fit(df).transform(df).show

// +---+----------------+--------------------+-------------------+
// | id| words| vector|UDF:indices(vector)|
// +---+----------------+--------------------+-------------------+
// | 1|[I, like, Spark]|(4,[0,1,3],[1.0,1...| [0, 1, 3]|
// | 2|[I, hate, Spark]|(4,[0,1,2],[1.0,1...| [0, 1, 2]|
// +---+----------------+--------------------+-------------------+

关于java - Spark : StringIndexer on sentences,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/47684543/

26 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com