scala - 由于类型不匹配(单元和字符串)，不能像这样在 reduce 中打印？

Question

我想打印一个文件中的内容，下面的代码是我如何做到的。

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.rdd.RDD

object SimpleSpark {
    def main(arg: Array[String]) = {
        val distFile = sc.textFile("/a/path/to/a/file")
        val aClass: MyClass = new MyClass()

        val mappedRDD = aClass.doStuff(distFile)
        mappedRDD.reduce( (a, b) => println(a) )
        // println( mappedRDD.reduce( (a, b) => a + b + "\n" ) )
        // mappedRDD.foreach(println)
}


class MyClass() {
    def doStuff(rdd: RDD[String]) : RDD[String] = {
        val field = "Hello!"
        rdd.map( x => field + x )
    }
}

我的问题是:

注释掉的两行代码工作正常，但是，此 mappedRDD.reduce( (a, b) => println(a) ) 行出现如下错误:

cliu@cliu-ubuntu:Apache-Spark$ sbt package
[info] Set current project to Simple Project (in build file:/home/cliu/Documents/github/Apache-Spark/)
[info] Compiling 1 Scala source to /home/cliu/Documents/github/Apache-Spark/target/scala-2.10/classes...
[error] /home/cliu/Documents/github/Apache-Spark/src/main/scala/SimpleSpark.scala:72: type mismatch;
[error]  found   : Unit
[error]  required: String
[error]         mappedRDD.reduce( (a, b) => println(a) )
[error]                                            ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 3 s, completed Dec 2, 2015 5:18:24 PM

为什么 mappedRDD.reduce( (a, b) => println(a) ) 不工作？

为什么我打印的是 Unit 而不是 Sting 的 typr？

Answer 1

这是因为 println(a) 不返回任何内容 (Unit)，而 reduce 实际上期待的是 String被退回。你应该只返回 a:

val reduced = mappedRDD.reduce( (a, b) => a )

然后您可以对 reduced 执行操作以打印您的 a:

reduced.foreach(println)