scala - 用Scala，Apache Spark编写的编译错误保存模型

Question

我正在运行Apache Spark提供的示例源代码来创建FPGrowth模型。我想保存模型以备将来使用，因此我写了这段代码的结尾行(model.save):

import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.mllib.fpm.FPGrowth
import org.apache.spark.mllib.util._
import org.apache.spark.rdd.RDD
import org.apache.spark.sql._
import java.io._
import scala.collection.mutable.Set


object App {


  def main(args: Array[String]) {


    val conf = new SparkConf().setAppName("prediction").setMaster("local[*]")
    val sc = new SparkContext(conf)

    val data = sc.textFile("FPFeatureSeries.txt")

    val transactions: RDD[Array[String]] = data.map(s => s.trim.split(' '))       
    val fpg = new FPGrowth()
          .setMinSupport(0.1)
          .setNumPartitions(10)
    val model = fpg.run(transactions)

    val minConfidence = 0.8
    model.generateAssociationRules(minConfidence).collect().foreach { rule =>
        if(rule.confidence>minConfidence){  
          println(
            rule.antecedent.mkString("[", ",", "]")
              + " => " + rule.consequent .mkString("[", ",", "]")


                 + ", " + rule.confidence)
             }
            }
    model.save(sc, "FPGrowthModel");


    }
 }

问题是我收到编译错误:值保存不是org.apache.spark.mllib.fpm.FPGrowth的成员

我尝试过包括库并从文档中复制确切的示例，但是我仍然遇到相同的错误。

我正在使用Spark 2.0.0和Scala 2.10。

Answer 1

我遇到过同样的问题。
用这个来保存模型

sc.parallelize(Seq(model), 1).saveAsObjectFile("path")

并加载模型

val linRegModel = sc.objectFile[LinearRegressionModel]("path").first()

这可能会有所帮助。
what-is-the-right-way-to-save-load-models-in-spark-pyspark