apache-spark - 如何使用 PySpark 保存 IDFmodel

Question

我用 PySpark 和 ipython notebook 制作了一个 IDFModel，如下所示:

from pyspark import SparkContext
from pyspark.mllib.feature import HashingTF
from pyspark.mllib.feature import IDF

hashingTF = HashingTF()   #this will be used with hashing later

txtdata_train = sc.wholeTextFiles("/home/ubuntu/folder").sortByKey() #this returns RDD of (filename, string) pairs for each file from the directory

split_data_train = txtdata_train.map(parse) #my parse function puts RDD in form I want

tf_train = hashingTF.transform(split_data_train) #creates term frequency sparse vectors for the training set

tf_train.cache()

idf_train = IDF().fit(tf_train)    #makes IDFmodel, THIS IS WHAT I WANT TO SAVE!!!

tfidf_train = idf_train.transform(tf_train)

这是基于本指南 https://spark.apache.org/docs/1.2.0/mllib-feature-extraction.html .我想保存这个模型，以便稍后在不同的笔记本中再次加载它。但是，没有信息如何做到这一点，我找到的最接近的是:

Save Apache Spark mllib model in python

但是当我在答案中尝试建议时

idf_train.save(sc, "/home/ubuntu/newfolder")

我得到错误代码

AttributeError: 'IDFModel' object has no attribute 'save'

是否有我遗漏的东西或无法解决 IDFModel 对象？谢谢!

Answer 1

我在 Scala/Java 中做了类似的事情。它似乎有效，但可能效率不高。这个想法是将文件作为序列化对象写入并稍后读回。祝你好运! :)

try {
  val fileOut:FileOutputStream  = new FileOutputStream(savePath+"/idf.jserialized");
  val out:ObjectOutputStream  = new ObjectOutputStream(fileOut);
  out.writeObject(idf);
  out.close();
  fileOut.close();
  System.out.println("\nSerialization Successful... Checkout your specified output file..\n");
} catch {
  case foe:FileNotFoundException => foe.printStackTrace()
  case ioe:IOException => ioe.printStackTrace()
}