java - 如何解决Spark流程序中的 "Error running job streaming job"？-6ren

java - 如何解决Spark流程序中的 "Error running job streaming job"？

转载作者：行者123 更新时间：2023-12-02 09:38:27

我在 Spark 上创建了一个 Master 和一个 Worker。然后我创建了一个 Spark 流作业并尝试提交它，但在 Master 上它显示了一长串 java 错误

使用此命令启动主控:

spark-class org.apache.spark.deploy.master.Master

用它来启动工作线程:

spark-class org.apache.spark.deploy.worker.Worker Spark://ip:port

为了提交 Spark 作业，尝试了具有不同参数的命令:

spark-submit --class com.rba.boston.SparkHome RBA-jar-with-dependency.jar
spark-submit --class com.rba.boston.SparkHome --master Spark://10.220.45.105:7077 --driver-memory 2G --driver-cores 2 --conf Spark.driver。 port=9998 --executor-memory 2G --deploy-mode cluster --total-executor-cores 4 RBA-jar-with-dependency.jar

spark-submit --class com.rba.boston.SparkHome --master Spark://10.220.45.105:7077 --driver-memory 2G --driver-cores 2 --conf Spark.driver。 port=9997 --conf Spark.driver.host=10.220.45.105 --executor-memory 2G --deploy-mode client --total-executor-cores 4 RBA-jar-with-dependency.jar

public static void main(String[] args) throws InterruptedException 
{
   SparkConf conf = new 
   SparkConf().setMaster("local[3]").setAppName("NetworkWordCount");
   JavaStreamingContext jssc = new JavaStreamingContext(conf, 
   Durations.seconds(3));
   JavaReceiverInputDStream<String> 
   lines=jssc.socketTextStream("10.220.45.105", 9998);
   JavaDStream<String> words = lines.flatMap(x -> 
   Arrays.asList(x.split(" ")).iterator());
   JavaPairDStream<String, Integer> pairs = words.mapToPair(s -> new 
   Tuple2<>(s, 1));
   JavaPairDStream<String, Integer> wordCounts = 
   pairs.reduceByKey((i1, i2) -> i1 + i2);
   wordCounts.print();
   jssc.start(); 
   jssc.awaitTermination();   // Wait for the computation to terminate
}

Logs after Submitting the job are:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/C:/Data/Softwares/spark-2.4.3-bin-hadoop2.6/jars/spark-unsafe_2.11-2.4.3.jar) to method java.nio.Bits.unaligned()
        WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
      WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
    WARNING: All illegal access operations will be denied in a future release
    SparkHome
    19/08/01 15:16:06 INFO SparkContext: Running Spark version 2.4.3
    19/08/01 15:16:06 INFO SparkContext: Submitted application: NetworkWordCount
    19/08/01 15:16:06 INFO SecurityManager: Changing view acls to: SachdeJ
    19/08/01 15:16:06 INFO SecurityManager: Changing modify acls to: SachdeJ
    19/08/01 15:16:06 INFO SecurityManager: Changing view acls groups to:
    19/08/01 15:16:06 INFO SecurityManager: Changing modify acls groups to:
    19/08/01 15:16:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(SachdeJ); groups with view permissions: Set(); users  with modify permissions: Set(SachdeJ); groups with modify permissions: Set()
    19/08/01 15:16:07 INFO Utils: Successfully started service 'sparkDriver' on port 54035.
    19/08/01 15:16:07 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
    19/08/01 15:16:07 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
    19/08/01 15:16:07 INFO DiskBlockManager: Created local directory at C:\Users\sachdej\AppData\Local\Temp\blockmgr-cccaa7a1-cfdc-45fb-8ff5-9a800def11ff
    19/08/01 15:16:07 INFO MemoryStore: MemoryStore started with capacity 434.4 MB
    19/08/01 15:16:08 INFO Utils: Successfully started service 'SparkUI' on port 4040.
    19/08/01 15:16:08 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://GGN1037742.bsci.bossci.com:4040
    19/08/01 15:16:08 INFO SparkContext: Added JAR file:/C:/Data/SharkTank/com.rba.boston/target/RBA-jar-with-dependencies.jar at spark://GGN1037742.bsci.bossci.com:54035/jars/RBA-jar-with-dependencies.jar with timestamp 1564652768180
    19/08/01 15:16:08 INFO Executor: Starting executor ID driver on host localhost
    19/08/01 15:16:08 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 54057.
    19/08/01 15:16:08 INFO NettyBlockTransferService: Server created on GGN1037742.bsci.bossci.com:54057
    19/08/01 15:16:08 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
    19/08/01 15:16:08 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, GGN1037742.bsci.bossci.com, 54057, None)
    19/08/01 15:16:08 INFO BlockManagerMasterEndpoint: Registering block manager GGN1037742.bsci.bossci.com:54057 with 434.4 MB RAM, BlockManagerId(driver, GGN1037742.bsci.bossci.com, 54057, None)
    19/08/01 15:16:08 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, GGN1037742.bsci.bossci.com, 54057, None)
    19/08/01 15:16:08 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, GGN1037742.bsci.bossci.com, 54057, None)
    19/08/01 15:16:08 INFO ReceiverTracker: Starting 1 receivers
    19/08/01 15:16:08 INFO ReceiverTracker: ReceiverTracker started
    19/08/01 15:16:08 INFO SocketInputDStream: Slide time = 3000 ms
    19/08/01 15:16:08 INFO SocketInputDStream: Storage level = Serialized 1x Replicated
    19/08/01 15:16:08 INFO SocketInputDStream: Checkpoint interval = null
    19/08/01 15:16:08 INFO SocketInputDStream: Remember interval = 3000 ms
    19/08/01 15:16:08 INFO SocketInputDStream: Initialized and validated org.apache.spark.streaming.dstream.SocketInputDStream@628a467a
    19/08/01 15:16:08 INFO FlatMappedDStream: Slide time = 3000 ms
    19/08/01 15:16:08 INFO FlatMappedDStream: Storage level = Serialized 1x Replicated
    19/08/01 15:16:08 INFO FlatMappedDStream: Checkpoint interval = null
    19/08/01 15:16:08 INFO FlatMappedDStream: Remember interval = 3000 ms
    19/08/01 15:16:08 INFO FlatMappedDStream: Initialized and validated org.apache.spark.streaming.dstream.FlatMappedDStream@4013fae0
    19/08/01 15:16:08 INFO MappedDStream: Slide time = 3000 ms
    19/08/01 15:16:08 INFO MappedDStream: Storage level = Serialized 1x Replicated
    19/08/01 15:16:08 INFO MappedDStream: Checkpoint interval = null
    19/08/01 15:16:08 INFO MappedDStream: Remember interval = 3000 ms
    19/08/01 15:16:08 INFO MappedDStream: Initialized and validated org.apache.spark.streaming.dstream.MappedDStream@2e5a119c
    19/08/01 15:16:08 INFO ShuffledDStream: Slide time = 3000 ms
    19/08/01 15:16:08 INFO ShuffledDStream: Storage level = Serialized 1x Replicated
    19/08/01 15:16:08 INFO ShuffledDStream: Checkpoint interval = null
    19/08/01 15:16:08 INFO ShuffledDStream: Remember interval = 3000 ms
    19/08/01 15:16:08 INFO ShuffledDStream: Initialized and validated org.apache.spark.streaming.dstream.ShuffledDStream@2141cddb
    19/08/01 15:16:08 INFO ForEachDStream: Slide time = 3000 ms
    19/08/01 15:16:08 INFO ForEachDStream: Storage level = Serialized 1x Replicated
    19/08/01 15:16:08 INFO ForEachDStream: Checkpoint interval = null
    19/08/01 15:16:08 INFO ForEachDStream: Remember interval = 3000 ms
    19/08/01 15:16:08 INFO ForEachDStream: Initialized and validated org.apache.spark.streaming.dstream.ForEachDStream@1148f61f
    19/08/01 15:16:08 INFO RecurringTimer: Started timer for JobGenerator at time 1564652769000
    19/08/01 15:16:08 INFO JobGenerator: Started JobGenerator at 1564652769000 ms
    19/08/01 15:16:08 INFO JobScheduler: Started JobScheduler
    19/08/01 15:16:08 INFO ReceiverTracker: Receiver 0 started
    19/08/01 15:16:08 INFO StreamingContext: StreamingContext started
    19/08/01 15:16:08 INFO DAGScheduler: Got job 0 (start at SparkHome.java:40) with 1 output partitions
    19/08/01 15:16:08 INFO DAGScheduler: Final stage: ResultStage 0 (start at SparkHome.java:40)
    19/08/01 15:16:08 INFO DAGScheduler: Parents of final stage: List()
    19/08/01 15:16:08 INFO DAGScheduler: Missing parents: List()
    19/08/01 15:16:09 INFO DAGScheduler: Submitting ResultStage 0 (Receiver 0 ParallelCollectionRDD[0] at makeRDD at ReceiverTracker.scala:614), which has no missing parents
    19/08/01 15:16:09 INFO JobScheduler: Added jobs for time 1564652769000 ms
    19/08/01 15:16:09 INFO JobScheduler: Starting job streaming job 1564652769000 ms.0 from job set of time 1564652769000 ms
    19/08/01 15:16:09 INFO JobScheduler: Finished job streaming job 1564652769000 ms.0 from job set of time 1564652769000 ms
    19/08/01 15:16:09 ERROR JobScheduler: Error running job streaming job 1564652769000 ms.0
    java.lang.IllegalArgumentException: Unsupported class file major version 56
            at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:166)
            at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:148)
            at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:136)
            at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:237)
            at org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:49)
            at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:517)
            at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:500)
            at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
            at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134)
            at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134)
            at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
            at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
            at scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:134)
            at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
            at org.apache.spark.util.FieldAccessFinder$$anon$3.visitMethodInsn(ClosureCleaner.scala:500)
            at org.apache.xbean.asm6.ClassReader.readCode(ClassReader.java:2175)
            at org.apache.xbean.asm6.ClassReader.readMethod(ClassReader.java:1238)
            at org.apache.xbean.asm6.ClassReader.accept(ClassReader.java:631)
            at org.apache.xbean.asm6.ClassReader.accept(ClassReader.java:355)
            at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:307)
            at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:306)
            at scala.collection.immutable.List.foreach(List.scala:392)
            at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:306)
            at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
            at org.apache.spark.SparkContext.clean(SparkContext.scala:2326)
            at org.apache.spark.SparkContext.runJob(SparkContext.scala:2100)
            at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1364)
            at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
            at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
            at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
            at org.apache.spark.rdd.RDD.take(RDD.scala:1337)
            at org.apache.spark.streaming.dstream.DStream$$anonfun$print$2$$anonfun$foreachFunc$3$1.apply(DStream.scala:735)
            at org.apache.spark.streaming.dstream.DStream$$anonfun$print$2$$anonfun$foreachFunc$3$1.apply(DStream.scala:734)
            at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
            at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
            at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
            at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
            at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
            at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
            at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
            at scala.util.Try$.apply(Try.scala:192)
            at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
            at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
            at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
            at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
            at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
            at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            at java.base/java.lang.Thread.run(Thread.java:835)
    Exception in thread "main" java.lang.IllegalArgumentException: Unsupported class file major version 56
            at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:166)
            at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:148)
            at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:136)
            at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:237)
            at org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:49)
            at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:517)
            at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:500)
            at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
            at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134)
            at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134)
            at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
            at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
            at scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:134)
            at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
            at org.apache.spark.util.FieldAccessFinder$$anon$3.visitMethodInsn(ClosureCleaner.scala:500)
            at org.apache.xbean.asm6.ClassReader.readCode(ClassReader.java:2175)
            at org.apache.xbean.asm6.ClassReader.readMethod(ClassReader.java:1238)
            at org.apache.xbean.asm6.ClassReader.accept(ClassReader.java:631)
            at org.apache.xbean.asm6.ClassReader.accept(ClassReader.java:355)
            at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:307)
            at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:306)
            at scala.collection.immutable.List.foreach(List.scala:392)
            at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:306)
            at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
            at org.apache.spark.SparkContext.clean(SparkContext.scala:2326)
            at org.apache.spark.SparkContext.runJob(SparkContext.scala:2100)
            at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1364)
            at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
            at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
            at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
            at org.apache.spark.rdd.RDD.take(RDD.scala:1337)
            at org.apache.spark.streaming.dstream.DStream$$anonfun$print$2$$anonfun$foreachFunc$3$1.apply(DStream.scala:735)
            at org.apache.spark.streaming.dstream.DStream$$anonfun$print$2$$anonfun$foreachFunc$3$1.apply(DStream.scala:734)
            at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
            at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
            at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
            at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
            at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
            at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
            at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
            at scala.util.Try$.apply(Try.scala:192)
            at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
            at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
            at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
            at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
            at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
            at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            at java.base/java.lang.Thread.run(Thread.java:835)
    19/08/01 15:16:09 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 46.6 KB, free 434.4 MB)
    19/08/01 15:16:09 INFO StreamingContext: Invoking stop(stopGracefully=false) from shutdown hook
    19/08/01 15:16:09 INFO ReceiverTracker: Sent stop signal to all 1 receivers
    19/08/01 15:16:09 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 16.0 KB, free 434.3 MB)
    19/08/01 15:16:09 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on GGN1037742.bsci.bossci.com:54057 (size: 16.0 KB, free: 434.4 MB)
    19/08/01 15:16:09 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1161
    19/08/01 15:16:09 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (Receiver 0 ParallelCollectionRDD[0] at makeRDD at ReceiverTracker.scala:614) (first 15 tasks are for partitions Vector(0))
    19/08/01 15:16:09 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
    19/08/01 15:16:09 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 8459 bytes)
    19/08/01 15:16:09 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
    19/08/01 15:16:09 INFO Executor: Fetching spark://GGN1037742.bsci.bossci.com:54035/jars/RBA-jar-with-dependencies.jar with timestamp 1564652768180
    19/08/01 15:16:09 INFO TransportClientFactory: Successfully created connection to GGN1037742.bsci.bossci.com/10.220.45.105:54035 after 31 ms (0 ms spent in bootstraps)
    19/08/01 15:16:09 INFO Utils: Fetching spark://GGN1037742.bsci.bossci.com:54035/jars/RBA-jar-with-dependencies.jar to C:\Users\sachdej\AppData\Local\Temp\spark-5b831314-0918-401a-88d8-1c6a135d554c\userFiles-99183ed0-5ef0-4d7d-8ea1-8e928e026847\fetchFileTemp6554652685231905365.tmp
    19/08/01 15:16:09 INFO Executor: Adding file:/C:/Users/sachdej/AppData/Local/Temp/spark-5b831314-0918-401a-88d8-1c6a135d554c/userFiles-99183ed0-5ef0-4d7d-8ea1-8e928e026847/RBA-jar-with-dependencies.jar to class loader
    19/08/01 15:16:09 INFO RecurringTimer: Started timer for BlockGenerator at time 1564652770000
    19/08/01 15:16:09 INFO BlockGenerator: Started BlockGenerator
    19/08/01 15:16:09 INFO BlockGenerator: Started block pushing thread
    19/08/01 15:16:09 INFO ReceiverSupervisorImpl: Stopping receiver with message: Registered unsuccessfully because Driver refused to start receiver 0:
    19/08/01 15:16:09 WARN ReceiverSupervisorImpl: Skip stopping receiver because it has not yet stared
    19/08/01 15:16:09 INFO BlockGenerator: Stopping BlockGenerator
    19/08/01 15:16:10 INFO RecurringTimer: Stopped timer for BlockGenerator after time 1564652770200
    19/08/01 15:16:10 INFO BlockGenerator: Waiting for block pushing thread to terminate
    19/08/01 15:16:10 INFO BlockGenerator: Pushing out the last 0 blocks
    19/08/01 15:16:10 INFO BlockGenerator: Stopped block pushing thread
    19/08/01 15:16:10 INFO BlockGenerator: Stopped BlockGenerator
    19/08/01 15:16:10 INFO ReceiverSupervisorImpl: Waiting for receiver to be stopped
    19/08/01 15:16:10 INFO ReceiverSupervisorImpl: Stopped receiver without error
    19/08/01 15:16:10 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 794 bytes result sent to driver
    19/08/01 15:16:10 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 965 ms on localhost (executor driver) (1/1)
    19/08/01 15:16:10 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
    19/08/01 15:16:10 INFO DAGScheduler: ResultStage 0 (start at SparkHome.java:40) finished in 1.229 s
    19/08/01 15:16:10 INFO ReceiverTracker: All of the receivers have deregistered successfully
    19/08/01 15:16:10 INFO ReceiverTracker: ReceiverTracker stopped
    19/08/01 15:16:10 INFO JobGenerator: Stopping JobGenerator immediately
    19/08/01 15:16:10 INFO RecurringTimer: Stopped timer for JobGenerator after time 1564652769000
    19/08/01 15:16:10 INFO JobGenerator: Stopped JobGenerator
    19/08/01 15:16:10 INFO JobScheduler: Stopped JobScheduler
    19/08/01 15:16:10 INFO StreamingContext: StreamingContext stopped successfully
    19/08/01 15:16:10 INFO SparkContext: Invoking stop() from shutdown hook
    19/08/01 15:16:10 INFO SparkUI: Stopped Spark web UI at http://GGN1037742.bsci.bossci.com:4040
    19/08/01 15:16:10 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
    19/08/01 15:16:10 INFO MemoryStore: MemoryStore cleared
    19/08/01 15:16:10 INFO BlockManager: BlockManager stopped
    19/08/01 15:16:10 INFO BlockManagerMaster: BlockManagerMaster stopped
    19/08/01 15:16:10 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
    19/08/01 15:16:10 INFO SparkContext: Successfully stopped SparkContext






    Master Logs after submitting the job are:
    19/08/01 14:03:10 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
    java.io.InvalidClassException: org.apache.spark.rpc.netty.NettyRpcEndpointRef; local class incompatible: stream classdesc serialVersionUID = -4186747031772874359, local class serialVersionUID = 6257082371135760434
            at java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689)
            at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1903)
            at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1772)
            at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2060)
            at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594)
            at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2355)
            at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2249)
            at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2087)
            at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594)
            at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
            at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
            at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:108)
            at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$deserialize$1$$anonfun$apply$1.apply(NettyRpcEnv.scala:271)
            at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
            at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:320)
            at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$deserialize$1.apply(NettyRpcEnv.scala:270)
            at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
            at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:269)
            at org.apache.spark.rpc.netty.RequestMessage$.apply(NettyRpcEnv.scala:611)
            at org.apache.spark.rpc.netty.NettyRpcHandler.internalReceive(NettyRpcEnv.scala:662)
            at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:654)
            at org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequestHandler.java:274)
            at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105)19/08/01 14:03:10 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
    java.io.InvalidClassException: org.apache.spark.rpc.netty.NettyRpcEndpointRef; local class incompatible: stream classdesc serialVersionUID = -4186747031772874359, local class serialVersionUID = 6257082371135760434
            at java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689)
            at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1903)
            at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1772)
            at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2060)
            at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594)
            at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2355)
            at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2249)
            at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2087)
            at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594)
            at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
            at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
            at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:108)
            at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$deserialize$1$$anonfun$apply$1.apply(NettyRpcEnv.scala:271)
            at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
            at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:320)
            at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$deserialize$1.apply(NettyRpcEnv.scala:270)
            at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
            at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:269)
            at org.apache.spark.rpc.netty.RequestMessage$.apply(NettyRpcEnv.scala:611)
            at org.apache.spark.rpc.netty.NettyRpcHandler.internalReceive(NettyRpcEnv.scala:662)
            at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:654)
            at org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequestHandler.java:274)
            at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105)19/08/01 14:03:10 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
    java.io.InvalidClassException: org.apache.spark.rpc.netty.NettyRpcEndpointRef; local class incompatible: stream classdesc serialVersionUID = -4186747031772874359, local class serialVersionUID = 6257082371135760434
            at java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689)
            at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1903)
            at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1772)
            at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2060)
            at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594)

最佳答案

类似的错误

java.lang.IllegalArgumentException: Unsupported class file major version 56

意味着该类是在比我们尝试运行它的版本更高的 Java 版本上编译的。

v56表示你使用的是Java 12编译，我不知道你的Spark使用的是哪个版本的Java。根据此票证，Spark 似乎还不兼容 Java 11:https://issues.apache.org/jira/browse/SPARK-24417

解决方案是使用以前版本的 jdk 编译代码。

关于java - 如何解决Spark流程序中的 "Error running job streaming job"？，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/57306839/

文章推荐： Java Socket 监听 PHP 客户端不工作

文章推荐： java - 执行功能文件时控制台不显示任何打印输出

文章推荐： java - 通过 JAXB 解析 XML 的可选命名空间

java - 流<流> : flatMap vs. 减少
这个问题在这里已经有了答案: Why filter() after flatMap() is "not completely" lazy in Java streams? (8 个答案) 关闭 6
instagram - 是否有像 Twitter 流 API 一样的 Instagram 流 API？
我正在创建一个应用程序来从 Instagram 收集数据。我正在寻找像 Twitter 流 API 这样的流 API，这样我就可以自动实时收集数据而无需发送请求。 Instagram 有类似的 API
java - 您的 InputStream 既不是 OLE2 流，也不是 OOXML 流
我正在使用 Apache Commons 在 Google App Engine 中上传一个 .docx 文件，如此链接中所述 File upload servlet .上传时，我还想使用 Apach
java - 使用 AWS Java DynamoDB 流 Kinesis 适配器处理 DynamoDB 流
我尝试使用 DynamoDB 流和 AWS 提供的 Java DynamoDB 流 Kinesis 适配器捕获 DynamoDB 表更改。我正在 Scala 应用程序中使用 AWS Java 开发工具
FFMPEG RTSP IP 流 (H.264) 转换为 RTSP 流 (MPEG-2)
我目前有一个采用 H.264 编码的 IP 摄像机流式视频 (RTSP)。我想使用 FFmpeg 将此 H.264 编码流转换为另一个 RTSP 流，但 MPEG-2 编码。我该怎么做？我应该使用哪
集群模式下的 Redis 流
Redis 流是否受益于集群模式？假设您有 10 个流，它们是分布在整个集群中还是都分布在同一节点上？我计划使用 Redis 流来实现真正的高吞吐量(200 万条消息/秒)，所以我担心这种规模的 Re
Java 流 - 在可空列表上进行流式处理的优雅方式
这件事困扰了我一段时间。所以我有一个 Product 类，它有一个 Image 列表(该列表可能为空)。我想做 product.getImages().stream().filter(...) 但
具有持久存储的 Redis 流
是否可以使用具有持久存储的 Redis 流还是流仅限于内存数据？我知道可以将 Redis 与核心数据结构的持久存储一起使用，但我已经能够理解是否也可以使用 Redis 中的流的持久存储。最佳答
elixir - 具有前瞻性的可枚举/流
我开始学习 Elixir 并遇到了一个我无法轻松解决的挑战。我正在尝试创建一个函数，该函数接受一个 Enumerable.t 并返回另一个 Enumerable.t ，其中包含下 n 个项目。它与
scala - readLines 流
我试图从 readLine 调用创建一个无限的字符串流: import java.io.{BufferedReader, InputStreamReader} val in = new Buffere
Java 8 流 : For Each
你能帮我使用 Java 8 流 API 编写以下代码吗？ SuperUser superUser = db.getSuperUser; for (final Client client : super
rust - 暗示AsyncRead为补品::流
我正在尝试服用补品routeguide tutorial，并将客户端变成rocket服务器。我只是接受响应并将gRPC转换为字符串。 service RouteGuide { rpc GetF
javascript - 流 - 类型可能与联合类型不兼容
流程代码可以是run here. 使用 flow，我有一个函数，它接受一个键值对对象并获取它的值 - 它获取的值应该是字符串、数字或 bool 值。 type ValueType = string
javascript - 流、对象字面量与联合不兼容
如果我有一个函数返回一个包含数据库信息的对象或一个空对象，如下所示: getThingFromDB: async function(id:string):Promise{ const from
facebook - 流。发布ogg
我正在尝试使用javascript api和FB.ui将ogg音频文件发布到流中，但是我不知道该怎么做。这是我给FB.ui的电话: FB.ui( { method: '
Accurev:无法删除工作区/流
我正在尝试删除工作区(或克隆它以使其看起来像父工作区，但我似乎两者都做不到)。但是，当我尝试时，我收到此消息:无法删除工作区 test_workspace，因为它有一个非空的默认组。据我所知，这意味
java - 流 'map'可以用于这样的处理吗？
可以使用 Stream|Map 来完成此操作，这样我就不需要将结果放入外部 HashMap 中，而是使用 .collect(Collectors.toMap(...)); 收集结果？ Map rep
Java 流 API
当我们从集合列表中获取 Stream 时，幕后到底发生了什么？我发现很多博客都说Stream不存储任何数据。如果这是真的，请考虑代码片段: List list = new ArrayList(); l
带列表的 Java 流
我对流及其工作方式不熟悉，我正在尝试获取列表中添加的特定对象的出现次数。我找到了一种使用Collections来做到这一点的方法。其过程如下: for (int i = 0; i p.conten
Java 流 - 映射列表到简化的映射列表
我希望将一个 map 列表转换为另一个分组的 map 列表。所以我有以下 map 列表 - List [{ "accId":"1", "accName":"TestAcc1", "accNumber

行者123

个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章

滴滴打车优惠券免费领取

全站热门文章

首页

博学

6Ren·AI

商城

java - 如何解决Spark流程序中的 "Error running job streaming job"？