scala - REPL 返回 RDD 值，但 SBT 无法编译

Question

当从新的 Spark shell REPL session 运行以下方法时，一切正常。但是，当我尝试编译包含此方法的类时，出现以下错误

Error:(21, 50) value values is not a member of org.apache.spark.rdd.RDD[(Long, org.apache.spark.mllib.recommendation.Rating)]
val training = ratings.filter(x => x._1 < 6).values.repartition(numPartitions).persist
                                             ^
Error:(22, 65) value values is not a member of org.apache.spark.rdd.RDD[(Long, org.apache.spark.mllib.recommendation.Rating)]
val validation = ratings.filter(x => x._1 >= 6 && x._1 < 8).values.repartition(numPartitions).persist
                                                            ^
Error:(23, 47) value values is not a member of org.apache.spark.rdd.RDD[(Long, org.apache.spark.mllib.recommendation.Rating)]
val test = ratings.filter(x => x._1 >= 8).values.persist
                                          ^

在这两种情况下，我都使用 Spark 1.0.1 代码本身如下。

def createDataset(ratings: RDD[Tuple2[Long,Rating]]): List[RDD[Rating]] = {

    val training = ratings.filter(x => x._1 < 6).values.repartition(numPartitions).persist
    val validation = ratings.filter(x => x._1 >= 6 && x._1 < 8).values.repartition(numPartitions).persist
    val test = ratings.filter(x => x._1 >= 8).values.persist
    val numTraining = training.count
    val numValidation = validation.count
    val numTest = test.count

    println(" Number Of Training ::: " + numTraining + " numValidation ::: " + numValidation + " ::: " + numTest)
    List(training,validation,test)
  }

它取自 MLLib 教程(稍作修改)，不知道出了什么问题。

Answer 1

您需要在代码中包含此行

import org.apache.spark.SparkContext._

这会将隐式转换导入 PairRDDFunctions这就是您可以调用值的原因。 Spark REPL 会为您导入此内容，这就是您在解释器中看不到错误的原因。具体来说，SparkContext 中的这个函数会执行转换。

implicit def rddToPairRDDFunctions[K, V](rdd: RDD[(K, V)])
      (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null) = {
    new PairRDDFunctions(rdd)
  }