gpt4 book ai didi

scala - Spark - 错误 : Failed to load class - spark-submit

转载 作者:行者123 更新时间:2023-12-05 06:52:59 25 4
gpt4 key购买 nike

我使用 Intellij 创建 sbt 项目并将工件构建到 jar 文件。我将 jar 文件放到服务器并提交,但出现此错误:

spark-submit --master spark://master:7077 --class streaming_process spark-jar/spark-streaming.jar

Error: Failed to load class streaming_process.
21/01/23 04:41:32 INFO ShutdownHookManager: Shutdown hook called
21/01/23 04:41:32 INFO ShutdownHookManager: Deleting directory /tmp/spark-982e8fe3-9421-45bd-aced-e46c4d756054

我的代码

// Code Block 2 Starts Here
val spark = SparkSession.builder
.master("spark://master:7077")
.appName("Stream Processing Application")
.getOrCreate()

spark.sparkContext.setLogLevel("ERROR")
// Code Block 2 Ends Here

// Code Block 3 Starts Here
// Stream meetup.com RSVP Message Data from Kafka
val meetup_rsvp_df = spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", kafka_bootstrap_servers)
.option("subscribe", kafka_topic_name)
.option("startingOffsets", "latest")
.load()

你可以看到我的项目图片:

enter image description here

最佳答案

JVM 找不到包含 streaming_process 类的 jar。请使用 --jars spark-jar/spark-streaming.jar 选项。

关于scala - Spark - 错误 : Failed to load class - spark-submit,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/65853202/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com