scala - 为什么创建表时出现 “Hive support is required to CREATE Hive TABLE (AS SELECT)”错误？

Question

我在尝试创建表时遇到问题。

以下是创建表的代码，其中发生了异常:

sparkSession.sql(s"CREATE TABLE IF NOT EXISTS mydatabase.students(" +
s"name string," + s"age int)")

这是 Spark session 配置:

lazy val sparkSession = SparkSession
.builder()
.appName("student_mapping")
.enableHiveSupport()
.getOrCreate()

这是一个异常(exception):

org.apache.spark.sql.AnalysisException: Hive support is required to 
CREATE Hive TABLE (AS SELECT);;'CreateTable `mydatabase`.`students`,
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Ignore

我的问题是:为什么会出现这个异常？我有几个其他 Spark 程序使用相同的 session 配置运行，运行完美。我使用的是 Scala 2.11 和 Spark 2.3。

Answer 1

SparkSession is the entry point to Spark SQL. It is one of the very first objects you create while developing a Spark SQL application.

SessionState is the state separation layer between Spark SQL sessions, including SQL configuration, tables, functions, UDFs, SQL parser, and everything else that depends on a SQLConf.

SessionState is available as the sessionState property of a SparkSession

Internally, sessionState clones the optional parent SessionState (if given when creating the SparkSession) or creates a new SessionState using BaseSessionStateBuilder as defined by spark.sql.catalogImplementation configuration property:

in-memory (default) for org.apache.spark.sql.internal.SessionStateBuilder

hive for org.apache.spark.sql.hive.HiveSessionStateBuilder

要使用hive，您应该使用类org.apache.spark.sql.hive.HiveSessionStateBuilder，根据文档，这可以通过设置属性spark.sql.catalogImplementation来完成创建SparkSession对象时到hive:

val conf = new SparkConf
      .set("spark.sql.warehouse.dir", "hdfs://namenode/sql/metadata/hive")
      .set("spark.sql.catalogImplementation","hive")
      .setMaster("local[*]")
      .setAppName("Hive Example")

val spark = SparkSession.builder()
      .config(conf)
      .enableHiveSupport()
      .getOrCreate()

或者您可以在将作业提交到集群时传递属性--conf spark.sql.catalogImplementation=hive。