apache-spark - 如何在本地模式下运行 Spark SQL Thrift Server 并使用 JDBC 连接到 Delta

Question

我想使用 JDBC 连接到 Delta，并想在本地模式下运行 Spark Thrift 服务器 (STS) 以启动轮胎。

我使用以下命令启动 STS:

$SPARK_HOME/sbin/start-thriftserver.sh \
  --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
  --packages 'io.delta:delta-core_2.12:1.0.0'

尝试使用直线连接时出现以下错误:

Beeline version 2.3.7 by Apache Hive
beeline> !connect jdbc:hive2://localhost:10000

Connecting to jdbc:hive2://localhost:10000
Enter username for jdbc:hive2://localhost:10000:
Enter password for jdbc:hive2://localhost:10000: 

Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: Failed to open new session: org.apache.spark.SparkException: Cannot find catalog plugin class for catalog 'spark_catalog': org.apache.spark.sql.delta.catalog.DeltaCatalog (state=08S01,code=0)

查看 thrift 服务器日志，我没有看到任何明显的错误，delta jar 文件的加载没有任何错误。

Spark Command: /Users/sandbox/.sdkman/candidates/java/8.0.282.j9-adpt/bin/java -cp /Users/sandbox/projects/delta/spark-3.1.2-bin-hadoop3.2/conf/:/Users/sandbox/projects/delta/spark-3.1.2-bin-hadoop3.2/jars/* -Xmx1g org.apache.spark.deploy.SparkSubmit --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name Thrift JDBC/ODBC Server --packages io.delta:delta-core_2.12:1.0.0 spark-internal
========================================
:: loading settings :: url = jar:file:/Users/sandbox/projects/delta/spark-3.1.2-bin-hadoop3.2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /Users/sandbox/.ivy2/cache
The jars for the packages stored in: /Users/sandbox/.ivy2/jars
io.delta#delta-core_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-8f65b233-668d-46ea-98d6-0da629b12d1e;1.0
        confs: [default]
        found io.delta#delta-core_2.12;1.0.0 in central
        found org.antlr#antlr4;4.7 in central
        found org.antlr#antlr4-runtime;4.7 in central
        found org.antlr#antlr-runtime;3.5.2 in central
        found org.antlr#ST4;4.0.8 in central
        found org.abego.treelayout#org.abego.treelayout.core;1.0.3 in central
        found org.glassfish#javax.json;1.0.4 in central
        found com.ibm.icu#icu4j;58.2 in central
:: resolution report :: resolve 385ms :: artifacts dl 13ms
        :: modules in use:
        com.ibm.icu#icu4j;58.2 from central in [default]
        io.delta#delta-core_2.12;1.0.0 from central in [default]
        org.abego.treelayout#org.abego.treelayout.core;1.0.3 from central in [default]
        org.antlr#ST4;4.0.8 from central in [default]
        org.antlr#antlr-runtime;3.5.2 from central in [default]
        org.antlr#antlr4;4.7 from central in [default]
        org.antlr#antlr4-runtime;4.7 from central in [default]
        org.glassfish#javax.json;1.0.4 from central in [default]
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
        ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-8f65b233-668d-46ea-98d6-0da629b12d1e
        confs: [default]
        0 artifacts copied, 8 already retrieved (0kB/13ms)
21/11/06 07:28:30 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/11/06 07:28:30 INFO HiveThriftServer2: Started daemon with process name: 75591@MacBook-Pro
21/11/06 07:28:30 INFO SignalUtils: Registering signal handler for TERM
21/11/06 07:28:30 INFO SignalUtils: Registering signal handler for HUP
21/11/06 07:28:30 INFO SignalUtils: Registering signal handler for INT
21/11/06 07:28:30 INFO HiveThriftServer2: Starting SparkContext
21/11/06 07:28:31 INFO HiveConf: Found configuration file null
21/11/06 07:28:31 INFO SparkContext: Running Spark version 3.1.2
21/11/06 07:28:31 INFO ResourceUtils: ==============================================================
21/11/06 07:28:31 INFO ResourceUtils: No custom resources configured for spark.driver.
21/11/06 07:28:31 INFO ResourceUtils: ==============================================================
21/11/06 07:28:31 INFO SparkContext: Submitted application: Thrift JDBC/ODBC Server
21/11/06 07:28:31 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
21/11/06 07:28:31 INFO ResourceProfile: Limiting resource is cpu
21/11/06 07:28:31 INFO ResourceProfileManager: Added ResourceProfile id: 0
...
21/11/06 07:28:31 INFO Utils: Successfully started service 'sparkDriver' on port 55507.
...
21/11/06 07:28:32 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/11/06 07:28:32 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://macbook-pro:4040
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/io.delta_delta-core_2.12-1.0.0.jar at spark://macbook-pro:55507/jars/io.delta_delta-core_2.12-1.0.0.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-4.7.jar at spark://macbook-pro:55507/jars/org.antlr_antlr4-4.7.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-runtime-4.7.jar at spark://macbook-pro:55507/jars/org.antlr_antlr4-runtime-4.7.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/org.antlr_antlr-runtime-3.5.2.jar at spark://macbook-pro:55507/jars/org.antlr_antlr-runtime-3.5.2.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/org.antlr_ST4-4.0.8.jar at spark://macbook-pro:55507/jars/org.antlr_ST4-4.0.8.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar at spark://macbook-pro:55507/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/org.glassfish_javax.json-1.0.4.jar at spark://macbook-pro:55507/jars/org.glassfish_javax.json-1.0.4.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added JAR file:///Users/sandbox/.ivy2/jars/com.ibm.icu_icu4j-58.2.jar at spark://macbook-pro:55507/jars/com.ibm.icu_icu4j-58.2.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/io.delta_delta-core_2.12-1.0.0.jar at file:///Users/sandbox/.ivy2/jars/io.delta_delta-core_2.12-1.0.0.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/io.delta_delta-core_2.12-1.0.0.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/io.delta_delta-core_2.12-1.0.0.jar
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-4.7.jar at file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-4.7.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/org.antlr_antlr4-4.7.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-4.7.jar
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-runtime-4.7.jar at file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-runtime-4.7.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/org.antlr_antlr4-runtime-4.7.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-runtime-4.7.jar
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/org.antlr_antlr-runtime-3.5.2.jar at file:///Users/sandbox/.ivy2/jars/org.antlr_antlr-runtime-3.5.2.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/org.antlr_antlr-runtime-3.5.2.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr-runtime-3.5.2.jar
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/org.antlr_ST4-4.0.8.jar at file:///Users/sandbox/.ivy2/jars/org.antlr_ST4-4.0.8.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/org.antlr_ST4-4.0.8.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_ST4-4.0.8.jar
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar at file:///Users/sandbox/.ivy2/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/org.glassfish_javax.json-1.0.4.jar at file:///Users/sandbox/.ivy2/jars/org.glassfish_javax.json-1.0.4.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/org.glassfish_javax.json-1.0.4.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.glassfish_javax.json-1.0.4.jar
21/11/06 07:28:32 INFO SparkContext: Added file file:///Users/sandbox/.ivy2/jars/com.ibm.icu_icu4j-58.2.jar at file:///Users/sandbox/.ivy2/jars/com.ibm.icu_icu4j-58.2.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: Copying /Users/sandbox/.ivy2/jars/com.ibm.icu_icu4j-58.2.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/com.ibm.icu_icu4j-58.2.jar
21/11/06 07:28:32 INFO Executor: Starting executor ID driver on host macbook-pro
21/11/06 07:28:32 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-runtime-4.7.jar with timestamp 1636183711298
21/11/06 07:28:32 INFO Utils: /Users/sandbox/.ivy2/jars/org.antlr_antlr4-runtime-4.7.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-runtime-4.7.jar
21/11/06 07:28:32 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/com.ibm.icu_icu4j-58.2.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: /Users/sandbox/.ivy2/jars/com.ibm.icu_icu4j-58.2.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/com.ibm.icu_icu4j-58.2.jar
21/11/06 07:28:33 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: /Users/sandbox/.ivy2/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar
21/11/06 07:28:33 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/org.glassfish_javax.json-1.0.4.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: /Users/sandbox/.ivy2/jars/org.glassfish_javax.json-1.0.4.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.glassfish_javax.json-1.0.4.jar
21/11/06 07:28:33 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/org.antlr_antlr-runtime-3.5.2.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: /Users/sandbox/.ivy2/jars/org.antlr_antlr-runtime-3.5.2.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr-runtime-3.5.2.jar
21/11/06 07:28:33 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/org.antlr_antlr4-4.7.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: /Users/sandbox/.ivy2/jars/org.antlr_antlr4-4.7.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-4.7.jar
21/11/06 07:28:33 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/org.antlr_ST4-4.0.8.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: /Users/sandbox/.ivy2/jars/org.antlr_ST4-4.0.8.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_ST4-4.0.8.jar
21/11/06 07:28:33 INFO Executor: Fetching file:///Users/sandbox/.ivy2/jars/io.delta_delta-core_2.12-1.0.0.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: /Users/sandbox/.ivy2/jars/io.delta_delta-core_2.12-1.0.0.jar has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/io.delta_delta-core_2.12-1.0.0.jar
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO TransportClientFactory: Successfully created connection to MacBook-Pro/192.168.1.207:55507 after 53 ms (0 ms spent in bootstraps)
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7179925225828426556.tmp
21/11/06 07:28:33 INFO Utils: /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7179925225828426556.tmp has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.abego.treelayout_org.abego.treelayout.core-1.0.3.jar to class loader
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/org.glassfish_javax.json-1.0.4.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/org.glassfish_javax.json-1.0.4.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp6953292115063484853.tmp
21/11/06 07:28:33 INFO Utils: /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp6953292115063484853.tmp has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.glassfish_javax.json-1.0.4.jar
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.glassfish_javax.json-1.0.4.jar to class loader
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/org.antlr_ST4-4.0.8.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/org.antlr_ST4-4.0.8.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7481622034847054241.tmp
21/11/06 07:28:33 INFO Utils: /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7481622034847054241.tmp has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_ST4-4.0.8.jar
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_ST4-4.0.8.jar to class loader
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/org.antlr_antlr4-runtime-4.7.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/org.antlr_antlr4-runtime-4.7.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp8137443686268894230.tmp
21/11/06 07:28:33 INFO Utils: /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp8137443686268894230.tmp has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-runtime-4.7.jar
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-runtime-4.7.jar to class loader
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/org.antlr_antlr4-4.7.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/org.antlr_antlr4-4.7.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7474775280611839356.tmp
21/11/06 07:28:33 INFO Utils: /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7474775280611839356.tmp has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-4.7.jar
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr4-4.7.jar to class loader
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/io.delta_delta-core_2.12-1.0.0.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/io.delta_delta-core_2.12-1.0.0.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7899823112197045994.tmp
21/11/06 07:28:33 INFO Utils: /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp7899823112197045994.tmp has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/io.delta_delta-core_2.12-1.0.0.jar
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/io.delta_delta-core_2.12-1.0.0.jar to class loader
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/org.antlr_antlr-runtime-3.5.2.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/org.antlr_antlr-runtime-3.5.2.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp1438945171356535536.tmp
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/org.antlr_antlr-runtime-3.5.2.jar to class loader
21/11/06 07:28:33 INFO Executor: Fetching spark://macbook-pro:55507/jars/com.ibm.icu_icu4j-58.2.jar with timestamp 1636183711298
21/11/06 07:28:33 INFO Utils: Fetching spark://macbook-pro:55507/jars/com.ibm.icu_icu4j-58.2.jar to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp836117221049857229.tmp
21/11/06 07:28:33 INFO Utils: /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/fetchFileTemp836117221049857229.tmp has been previously copied to /private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/com.ibm.icu_icu4j-58.2.jar
21/11/06 07:28:33 INFO Executor: Adding file:/private/var/folders/qp/g1z81/T/spark-1270d/userFiles-2c8e44aa/com.ibm.icu_icu4j-58.2.jar to class loader
21/11/06 07:28:33 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 55509.
21/11/06 07:28:33 INFO NettyBlockTransferService: Server created on macbook-pro:55509
...
21/11/06 07:28:33 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/sandbox/projects/delta/spark-warehouse/').
21/11/06 07:28:33 INFO SharedState: Warehouse path is 'file:/Users/sandbox/projects/delta/spark-warehouse/'.
21/11/06 07:28:34 INFO HiveUtils: Initializing HiveMetastoreConnection version 2.3.7 using Spark classes.
21/11/06 07:28:35 INFO HiveConf: Found configuration file null
21/11/06 07:28:35 INFO SessionState: Created HDFS directory: /tmp/hive/sandbox/eae25990-9fbc-48c4-abe3-425df7d94eda
21/11/06 07:28:35 INFO SessionState: Created local directory: /var/folders/qp/g1z81/T/sandbox/eae25990-9fbc-48c4-abe3-425df7d94eda
21/11/06 07:28:35 INFO SessionState: Created HDFS directory: /tmp/hive/sandbox/eae25990-9fbc-48c4-abe3-425df7d94eda/_tmp_space.db
21/11/06 07:28:35 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is file:/Users/sandbox/projects/delta/spark-warehouse/
21/11/06 07:28:36 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
21/11/06 07:28:36 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
21/11/06 07:28:36 INFO HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
21/11/06 07:28:36 INFO ObjectStore: ObjectStore, initialize called
21/11/06 07:28:36 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
21/11/06 07:28:36 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
21/11/06 07:28:37 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
21/11/06 07:28:39 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
21/11/06 07:28:39 INFO ObjectStore: Initialized ObjectStore
21/11/06 07:28:39 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
21/11/06 07:28:39 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore sandbox@192.168.1.207
21/11/06 07:28:39 INFO HiveMetaStore: Added admin role in metastore
21/11/06 07:28:39 INFO HiveMetaStore: Added public role in metastore
21/11/06 07:28:39 INFO HiveMetaStore: No user is added in admin role, since config is empty
21/11/06 07:28:39 INFO HiveMetaStore: 0: get_all_functions
21/11/06 07:28:39 INFO audit: ugi=sandbox       ip=unknown-ip-addr      cmd=get_all_functions
21/11/06 07:28:39 INFO HiveMetaStore: 0: get_database: default
21/11/06 07:28:39 INFO audit: ugi=sandbox       ip=unknown-ip-addr      cmd=get_database: default      
21/11/06 07:28:39 INFO HiveUtils: Initializing execution hive, version 2.3.7
21/11/06 07:28:39 INFO SessionState: Created HDFS directory: /tmp/hive/sandbox/0911a389-88fd-467c-97a4-6d3f8ae8775d
21/11/06 07:28:39 INFO SessionState: Created local directory: /var/folders/qp/g1z81/T/sandbox/0911a389-88fd-467c-97a4-6d3f8ae8775d
21/11/06 07:28:39 INFO SessionState: Created HDFS directory: /tmp/hive/sandbox/0911a389-88fd-467c-97a4-6d3f8ae8775d/_tmp_space.db
21/11/06 07:28:39 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is file:/Users/sandbox/projects/delta/spark-warehouse/
21/11/06 07:28:40 INFO SessionManager: Operation log root directory is created: /var/folders/qp/g1z81/T/sandbox/operation_logs
...
21/11/06 07:28:40 INFO AbstractService: Service:OperationManager is inited.
21/11/06 07:28:40 INFO AbstractService: Service:SessionManager is inited.
21/11/06 07:28:40 INFO AbstractService: Service: CLIService is inited.
21/11/06 07:28:40 INFO AbstractService: Service:ThriftBinaryCLIService is inited.
21/11/06 07:28:40 INFO AbstractService: Service: HiveServer2 is inited.
21/11/06 07:28:40 INFO AbstractService: Service:OperationManager is started.
21/11/06 07:28:40 INFO AbstractService: Service:SessionManager is started.
21/11/06 07:28:40 INFO AbstractService: Service: CLIService is started.
21/11/06 07:28:40 INFO AbstractService: Service:ThriftBinaryCLIService is started.
21/11/06 07:28:40 INFO ThriftCLIService: Starting ThriftBinaryCLIService on port 10000 with 5...500 worker threads
21/11/06 07:28:40 INFO AbstractService: Service:HiveServer2 is started.
21/11/06 07:28:40 INFO HiveThriftServer2: HiveThriftServer2 started
...
21/11/06 07:29:48 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
21/11/06 07:29:48 INFO ObjectStore: Initialized ObjectStore
21/11/06 07:29:48 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
21/11/06 07:29:48 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore sandbox@192.168.1.207
21/11/06 07:29:48 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
21/11/06 07:29:49 INFO HiveMetaStore: Added admin role in metastore
21/11/06 07:29:49 INFO HiveMetaStore: Added public role in metastore
21/11/06 07:29:49 INFO HiveMetaStore: No user is added in admin role, since config is empty
21/11/06 07:29:49 INFO HiveMetaStore: 0: get_all_functions
21/11/06 07:29:49 INFO audit: ugi=sandbox       ip=unknown-ip-addr      cmd=get_all_functions  
21/11/06 07:29:49 INFO SessionState: Created HDFS directory: /tmp/hive/anonymous/a8fabb9a-68ed-42f3-b4a0-86cbac11b27d
21/11/06 07:29:49 INFO SessionState: Created local directory: /var/folders/qp/g1z81/T/sandbox/a8fabb9a-68ed-42f3-b4a0-86cbac11b27d
21/11/06 07:29:49 INFO SessionState: Created HDFS directory: /tmp/hive/anonymous/a8fabb9a-68ed-42f3-b4a0-86cbac11b27d/_tmp_space.db
21/11/06 07:29:49 INFO HiveSessionImpl: Operation log session directory is created: /var/folders/qp/g1z81/T/sandbox/operation_logs/a8fabb9a-68ed-42f3-b4a0-86cbac11b27d
ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.8ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.821/11/06 07:29:50 INFO SessionState: Deleted directory: /tmp/hive/anonymous/a8fabb9a-68ed-42f3-b4a0-86cbac11b27d on fs with scheme file
21/11/06 07:29:50 INFO SessionState: Deleted directory: /var/folders/qp/g1z81/T/sandbox/a8fabb9a-68ed-42f3-b4a0-86cbac11b27d on fs with scheme file
21/11/06 07:29:50 INFO HiveMetaStore: 0: Cleaning up thread local RawStore...
21/11/06 07:29:50 INFO audit: ugi=anonymous     ip=unknown-ip-addr      cmd=Cleaning up thread local RawStore...
21/11/06 07:29:50 INFO HiveMetaStore: 0: Done cleaning up thread local RawStore
21/11/06 07:29:50 INFO audit: ugi=anonymous     ip=unknown-ip-addr      cmd=Done cleaning up thread local RawStore     
21/11/06 07:29:50 WARN ThriftCLIService: Error opening session:
org.apache.hive.service.cli.HiveSQLException: Failed to open new session: org.apache.spark.SparkException: Cannot find catalog plugin class for catalog 'spark_catalog': org.apache.spark.sql.delta.catalog.DeltaCatalog
        at org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:85)
        at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:204)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:371)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:243)
        at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1497)
        at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1482)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:823)
Caused by: org.apache.spark.SparkException: Cannot find catalog plugin class for catalog 'spark_catalog': org.apache.spark.sql.delta.catalog.DeltaCatalog
        at org.apache.spark.sql.connector.catalog.Catalogs$.load(Catalogs.scala:66)
        at org.apache.spark.sql.connector.catalog.CatalogManager.loadV2SessionCatalog(CatalogManager.scala:66)
        at org.apache.spark.sql.connector.catalog.CatalogManager.$anonfun$v2SessionCatalog$2(CatalogManager.scala:85)
        at org.apache.spark.sql.connector.catalog.CatalogManager$$Lambda$1357/0x0000000000000000.apply(Unknown Source)
        at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
        at org.apache.spark.sql.connector.catalog.CatalogManager.$anonfun$v2SessionCatalog$1(CatalogManager.scala:85)
        at org.apache.spark.sql.connector.catalog.CatalogManager$$Lambda$1356/0x0000000000000000.apply(Unknown Source)
        at scala.Option.map(Option.scala:230)
        at org.apache.spark.sql.connector.catalog.CatalogManager.v2SessionCatalog(CatalogManager.scala:84)
        at org.apache.spark.sql.connector.catalog.CatalogManager.catalog(CatalogManager.scala:50)
        at org.apache.spark.sql.connector.catalog.CatalogManager.currentCatalog(CatalogManager.scala:117)
        at org.apache.spark.sql.connector.catalog.LookupCatalog.currentCatalog(LookupCatalog.scala:35)
        at org.apache.spark.sql.connector.catalog.LookupCatalog.currentCatalog$(LookupCatalog.scala:35)
...

我使用 Java 8 和 Spark 3.1.2，预构建 Hadoop 3.2 及更高版本和 Delta 1.0.0，在 Mac OSX 上运行

可能是什么问题？

Answer 1

一旦您可以将 io.delta:delta-core_2.12:1.0.0 JAR 文件复制到 $SPARK_HOME/lib 并重新启动，此错误就会消失。