Azure HDI Spark导入sqlContext.implicits._错误

Question

我在将数据从 Azure Blob 存储 csv 文件导入到我的 Spark by Jupyter 笔记本时遇到问题。我正在尝试实现关于 ML 和 Spark 的教程之一。当我像这样填写 Jupyter 笔记本时:

import sqlContext.implicits._
val flightDelayTextLines = sc.textFile("wasb://sparkcontainer@[my account].blob.core.windows.net/sparkcontainer/Scored_FlightsAndWeather.csv")

case class AirportFlightDelays(OriginAirportCode:String,OriginLatLong:String,Month:Integer,Day:Integer,Hour:Integer,Carrier:String,DelayPredicted:Integer,DelayProbability:Double)

val flightDelayRowsWithoutHeader = flightDelayTextLines.map(s => s.split(",")).filter(line => line(0) != "OriginAirportCode")

val resultDataFrame = flightDelayRowsWithoutHeader.map(
    s => AirportFlightDelays(
        s(0), //Airport code
        s(13) + "," + s(14), //Lat,Long
        s(1).toInt, //Month
        s(2).toInt, //Day
        s(3).toInt, //Hour
        s(5), //Carrier
        s(11).toInt, //DelayPredicted
        s(12).toDouble //DelayProbability
        )
).toDF()

resultDataFrame.write.mode("overwrite").saveAsTable("FlightDelays") 

我收到这样的错误:

SparkSession available as 'spark'.
<console>:23: error: not found: value sqlContext
       import sqlContext.implicits._
              ^

我也使用了短路径，例如(“wasb:///sparkcontainer/Scored_FlightsAndWeather.csv”)这个相同的错误。有任何想法吗？BR,马立克

Answer 1

当我看到您的代码片段时，我没有看到创建 sqlContext，请引用以下代码并创建 sqlContext，然后开始使用它。

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._