apache-spark - 使用 Spark 从 KMeans 的文本数据生成向量

Question

我是 Spark 和机器学习新手。我正在尝试使用 KMeans 进行聚类一些数据，例如

1::Hi How are you
2::I am fine, how about you

在数据中，分隔符是::，要聚类的实际文本是包含文本数据的第二列。在阅读了 Spark 官方页面和大量文章后，我编写了以下代码，但我无法生成向量作为 KMeans.train 步骤的输入。

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.mllib.clustering.{KMeans, KMeansModel}
import org.apache.spark.mllib.linalg.Vectors

val sc = new SparkContext("local", "test") 

val sqlContext= new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._

import org.apache.spark.ml.feature.{HashingTF, IDF, Tokenizer}

val rawData = sc.textFile("data/mllib/KM.txt").map(line => line.split("::")(1))

val sentenceData = rawData.toDF("sentence")

val tokenizer = new Tokenizer().setInputCol("sentence").setOutputCol("words")

val wordsData = tokenizer.transform(sentenceData)

val hashingTF = new HashingTF().setInputCol("words").setOutputCol("rawFeatures").setNumFeatures(20)

val featurizedData = hashingTF.transform(wordsData)

val clusters = KMeans.train(featurizedData, 2, 10)

我收到以下错误

<console>:27: error: type mismatch;
 found   : org.apache.spark.sql.DataFrame
    (which expands to)  org.apache.spark.sql.Dataset[org.apache.spark.sql.Row]
 required: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector]
       val clusters = KMeans.train(featurizedData, 2, 10)

请建议如何处理 KMeans 的输入数据

提前致谢。

Answer 1

最后我在替换以下代码后让它工作。

val hashingTF = new HashingTF().setInputCol("words").setOutputCol("rawFeatures").setNumFeatures(20)

val featurizedData = hashingTF.transform(wordsData)

val clusters = KMeans.train(featurizedData, 2, 10)

与

val hashingTF = new HashingTF().setNumFeatures(1000).setInputCol(tokenizer.getOutputCol).setOutputCol("features")

val kmeans = new KMeans().setK(2).setFeaturesCol("features").setPredictionCol("prediction")

val pipeline = new Pipeline().setStages(Array(tokenizer, hashingTF, kmeans))