scala - 如何使用IntelliJ上的Scala从spark连接到Hive？

Question

我是hive和spark的新手，正试图找出一种方法来访问hive中的表以操纵和访问数据。怎么做到呢？

Answer 1

Spark <2.0

 val sc = new SparkContext(conf)

 val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
 val myDataFrame = sqlContext.sql("select * from mydb.mytable")

在更高版本的spark中，使用SparkSession:

SparkSession is now the new entry point of Spark that replaces the old SQLContext and HiveContext. Note that the old SQLContext and HiveContext are kept for backward compatibility. A new catalog interface is accessible from SparkSession - existing API on databases and tables access such as listTables, createExternalTable, dropTempView, cacheTable are moved here. -- from the docs

val spark = SparkSession
  .builder()
  .appName("Spark Hive Example")
  .config("spark.sql.warehouse.dir", warehouseLocation)
  .enableHiveSupport()
  .getOrCreate()
 val myDataFrame = spark.sql("select * from mydb.mytable")