scala - 在 Scala 中导入 spark.implicits._

Question

我正在尝试导入 spark.implicits._
显然，这是 Scala 类中的一个对象。
当我以这样的方法导入它时:

def f() = {
  val spark = SparkSession()....
  import spark.implicits._
}

它工作正常，但是我正在编写一个测试类，我想让这个导入可用于所有测试
我试过了:

class SomeSpec extends FlatSpec with BeforeAndAfter {
  var spark:SparkSession = _

  //This won't compile
  import spark.implicits._

  before {
    spark = SparkSession()....
    //This won't either
    import spark.implicits._
  }

  "a test" should "run" in {
    //Even this won't compile (although it already looks bad here)
    import spark.implicits._

    //This was the only way i could make it work
    val spark = this.spark
    import spark.implicits._
  }
}

这不仅看起来很糟糕，我不想每次测试都这样做
什么是“正确”的做法？

Answer 1

您可以执行类似于 Spark 测试套件中所做的操作。例如，这会起作用(受 SQLTestData 启发):

class SomeSpec extends FlatSpec with BeforeAndAfter { self =>

  var spark: SparkSession = _

  private object testImplicits extends SQLImplicits {
    protected override def _sqlContext: SQLContext = self.spark.sqlContext
  }
  import testImplicits._

  before {
    spark = SparkSession.builder().master("local").getOrCreate()
  }

  "a test" should "run" in {
    // implicits are working
    val df = spark.sparkContext.parallelize(List(1,2,3)).toDF()
  }
}

或者你可以使用类似 SharedSQLContext 的东西直接，它提供了一个 testImplicits: SQLImplicits ， IE。:

class SomeSpec extends FlatSpec with SharedSQLContext {
  import testImplicits._

  // ...

}