java - Scala-Spark : save outcome into a file

Question

我正在使用 Scala API 在 Spark 中运行序列模型。这是查看结果的代码行:

model.freqSequences.collect().foreach { freqSequence => println(freqSequence.sequence.map(_.mkString("[", ", ", "]")).mkString("[", ", ", "]") + ", " + freqSequence.freq)}

问题是结果越来越大，不想再使用collect()，而是将其保存在HDFS或本地的文件中。我试过这个:

scala> val outcome = model.freqSequences.foreach { freqSequence => println(freqSequence.sequence.map(_.mkString("[", ", ", "]")).mkString("[", ", ", "]") + ", " + freqSequence.freq)}

scala> outcome.saveAsTextFile("tmp/outcome1/")

error: saveAsTextFile is not a member of Unit

结果是一个单位，我无法使用 saveAsTextFile。还有什么办法可以挽救这个结果吗？发送。

Answer 1

foreach 返回一个 Unit。

您希望首先映射到String，以便可以另存为文件。像这样的东西:

val outcome = model.freqSequences.map { freqSequence => freqSequence.sequence.map(_.mkString("[", ", ", "]")).mkString("[", ", ", "]") + ", " + freqSequence.freq}
// print
outcome.foreach(println)
// save
outcome.saveAsTextFile("tmp/outcome1/")