java - 未找到类 Spark Streaming 和 kafka-6ren

java - 未找到类 Spark Streaming 和 kafka

转载作者：行者123 更新时间：2023-12-02 04:13:35

我尝试使用 Spark Streaming 和 Kafka，但是当它收到消息时会引发错误。当它没有收到消息时，没有错误。我无法找出错误原因。详情请见下文:

Java

package org.cosmob.ccc.test;

import java.io.Serializable;
import java.util.HashMap;
import java.util.Map;

import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaPairDStream;
import org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import org.apache.spark.streaming.kafka.KafkaUtils;
import org.cosmob.ccc.spark.SparkConnect;
import org.apache.spark.streaming.Duration;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.PairFunction;
import scala.Tuple2;
import java.util.regex.Pattern;
import com.google.common.collect.Lists;

public class Test implements Serializable{
    private static final long serialVersionUID = -6483717103605642290L;

    private SparkConnect sparkConnect;
    private JavaStreamingContext jssc;

    private static final Pattern SPACE = Pattern.compile(" ");

    public Test() {

        sparkConnect = new SparkConnect();
        jssc = new JavaStreamingContext(sparkConnect.getCtx(), new Duration(1000));
    }

    public JavaStreamingContext getStreamContext(){
        return jssc;
    }

    public static void main(String[] args) {
        Test tt = new Test();

        String zkQuorum="172.16.201.173";
        String topic = "test";
        String group = "me";
        int numThreads = 1;
        Map<String, Integer> topicMap = new HashMap<String, Integer>();
        topicMap.put(topic, numThreads);

        JavaPairReceiverInputDStream<String, String> messages =
                KafkaUtils.createStream(tt.getStreamContext(), zkQuorum, group, topicMap);

        JavaDStream<String> lines = messages.map(new Function<Tuple2<String, String>, String>() {
            private static final long serialVersionUID = 1L;

            @Override
              public String call(Tuple2<String, String> tuple2) {
                return tuple2._2();
              }

            });

        lines.print();
        tt.getStreamContext().start();
        tt.getStreamContext().awaitTermination();
    }
}

错误

15/11/05 15:04:02 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
-------------------------------------------
Time: 1446725042000 ms
-------------------------------------------

15/11/05 15:04:02 INFO scheduler.JobScheduler: Added jobs for time 1446725042000 ms
15/11/05 15:04:02 INFO scheduler.JobScheduler: Starting job streaming job 1446725042000 ms.0 from job set of time 1446725042000 ms
15/11/05 15:04:02 INFO scheduler.JobScheduler: Finished job streaming job 1446725042000 ms.0 from job set of time 1446725042000 ms
15/11/05 15:04:02 INFO scheduler.JobScheduler: Total delay: 0.006 s for time 1446725042000 ms (execution: 0.000 s)
15/11/05 15:04:02 INFO rdd.MappedRDD: Removing RDD 23 from persistence list
15/11/05 15:04:02 INFO storage.BlockManager: Removing RDD 23
15/11/05 15:04:02 INFO rdd.BlockRDD: Removing RDD 22 from persistence list
15/11/05 15:04:02 INFO storage.BlockManager: Removing RDD 22
15/11/05 15:04:02 INFO kafka.KafkaInputDStream: Removing blocks of RDD BlockRDD[22] at BlockRDD at ReceiverInputDStream.scala:69 of time 1446725042000 ms
15/11/05 15:04:03 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
-------------------------------------------
Time: 1446725043000 ms
-------------------------------------------

15/11/05 15:04:03 INFO scheduler.JobScheduler: Added jobs for time 1446725043000 ms
15/11/05 15:04:03 INFO scheduler.JobScheduler: Starting job streaming job 1446725043000 ms.0 from job set of time 1446725043000 ms
15/11/05 15:04:03 INFO scheduler.JobScheduler: Finished job streaming job 1446725043000 ms.0 from job set of time 1446725043000 ms
15/11/05 15:04:03 INFO scheduler.JobScheduler: Total delay: 0.004 s for time 1446725043000 ms (execution: 0.000 s)
15/11/05 15:04:03 INFO rdd.MappedRDD: Removing RDD 25 from persistence list
15/11/05 15:04:03 INFO storage.BlockManager: Removing RDD 25
15/11/05 15:04:03 INFO rdd.BlockRDD: Removing RDD 24 from persistence list
15/11/05 15:04:03 INFO storage.BlockManager: Removing RDD 24
15/11/05 15:04:03 INFO kafka.KafkaInputDStream: Removing blocks of RDD BlockRDD[24] at BlockRDD at ReceiverInputDStream.scala:69 of time 1446725043000 ms
15/11/05 15:04:03 INFO storage.BlockManagerInfo: Added input-0-1446725042800 in memory on qmic-tag-s05.quwic.org:53916 (size: 77.0 B, free: 258.8 MB)
15/11/05 15:04:03 INFO storage.MemoryStore: ensureFreeSpace(77) called with curMem=13135, maxMem=996566630
15/11/05 15:04:03 INFO storage.MemoryStore: Block input-0-1446725042800 stored as bytes in memory (estimated size 77.0 B, free 950.4 MB)
15/11/05 15:04:03 INFO storage.BlockManagerInfo: Added input-0-1446725042800 in memory on wassimd.QUWIC.LOCAL:62993 (size: 77.0 B, free: 950.4 MB)
15/11/05 15:04:03 INFO storage.BlockManagerMaster: Updated info of block input-0-1446725042800
15/11/05 15:04:04 INFO scheduler.ReceiverTracker: Stream 0 received 1 blocks
15/11/05 15:04:04 INFO scheduler.JobScheduler: Added jobs for time 1446725044000 ms
15/11/05 15:04:04 INFO scheduler.JobScheduler: Starting job streaming job 1446725044000 ms.0 from job set of time 1446725044000 ms
15/11/05 15:04:04 INFO spark.SparkContext: Starting job: take at DStream.scala:608
15/11/05 15:04:04 INFO scheduler.DAGScheduler: Got job 3 (take at DStream.scala:608) with 1 output partitions (allowLocal=true)
15/11/05 15:04:04 INFO scheduler.DAGScheduler: Final stage: Stage 4(take at DStream.scala:608)
15/11/05 15:04:04 INFO scheduler.DAGScheduler: Parents of final stage: List()
15/11/05 15:04:04 INFO scheduler.DAGScheduler: Missing parents: List()
15/11/05 15:04:04 INFO scheduler.DAGScheduler: Submitting Stage 4 (MappedRDD[29] at map at MappedDStream.scala:35), which has no missing parents
15/11/05 15:04:04 INFO storage.MemoryStore: ensureFreeSpace(1872) called with curMem=13212, maxMem=996566630
15/11/05 15:04:04 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 1872.0 B, free 950.4 MB)
15/11/05 15:04:04 INFO storage.MemoryStore: ensureFreeSpace(1260) called with curMem=15084, maxMem=996566630
15/11/05 15:04:04 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 1260.0 B, free 950.4 MB)
15/11/05 15:04:04 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on wassimd.QUWIC.LOCAL:62993 (size: 1260.0 B, free: 950.4 MB)
15/11/05 15:04:04 INFO storage.BlockManagerMaster: Updated info of block broadcast_4_piece0
15/11/05 15:04:04 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 4 (MappedRDD[29] at map at MappedDStream.scala:35)
15/11/05 15:04:04 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
15/11/05 15:04:04 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 75, qmic-tag-s05.quwic.org, NODE_LOCAL, 1071 bytes)
15/11/05 15:04:04 INFO storage.BlockManagerInfo: Added input-0-1446725043800 in memory on qmic-tag-s05.quwic.org:53916 (size: 78.0 B, free: 258.8 MB)
15/11/05 15:04:04 INFO storage.MemoryStore: ensureFreeSpace(78) called with curMem=16344, maxMem=996566630
15/11/05 15:04:04 INFO storage.MemoryStore: Block input-0-1446725043800 stored as bytes in memory (estimated size 78.0 B, free 950.4 MB)
15/11/05 15:04:04 INFO storage.BlockManagerInfo: Added input-0-1446725043800 in memory on wassimd.QUWIC.LOCAL:62993 (size: 78.0 B, free: 950.4 MB)
15/11/05 15:04:04 INFO storage.BlockManagerMaster: Updated info of block input-0-1446725043800
15/11/05 15:04:04 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on qmic-tag-s05.quwic.org:53916 (size: 1260.0 B, free: 258.8 MB)
15/11/05 15:04:04 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 4.0 (TID 75, qmic-tag-s05.quwic.org): java.lang.ClassNotFoundException: org.cosmob.ccc.test.Test$1
        java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        java.security.AccessController.doPrivileged(Native Method)
        java.net.URLClassLoader.findClass(URLClassLoader.java:360)
        java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        java.lang.Class.forName0(Native Method)
        java.lang.Class.forName(Class.java:344)
        org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59)
        java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
        java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
        org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
        org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
        org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
        org.apache.spark.scheduler.Task.run(Task.scala:54)
        org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        java.lang.Thread.run(Thread.java:745)
15/11/05 15:04:04 INFO scheduler.TaskSetManager: Starting task 0.1 in stage 4.0 (TID 76, qmic-tag-s05.quwic.org, NODE_LOCAL, 1071 bytes)
15/11/05 15:04:04 INFO scheduler.TaskSetManager: Lost task 0.1 in stage 4.0 (TID 76) on executor qmic-tag-s05.quwic.org: java.lang.ClassNotFoundException (org.cosmob.ccc.test.Test$1) [duplicate 1]
15/11/05 15:04:04 INFO scheduler.TaskSetManager: Starting task 0.2 in stage 4.0 (TID 77, qmic-tag-s05.quwic.org, NODE_LOCAL, 1071 bytes)
15/11/05 15:04:04 INFO scheduler.TaskSetManager: Lost task 0.2 in stage 4.0 (TID 77) on executor qmic-tag-s05.quwic.org: java.lang.ClassNotFoundException (org.cosmob.ccc.test.Test$1) [duplicate 2]
15/11/05 15:04:04 INFO scheduler.TaskSetManager: Starting task 0.3 in stage 4.0 (TID 78, qmic-tag-s05.quwic.org, NODE_LOCAL, 1071 bytes)
15/11/05 15:04:04 INFO scheduler.TaskSetManager: Lost task 0.3 in stage 4.0 (TID 78) on executor qmic-tag-s05.quwic.org: java.lang.ClassNotFoundException (org.cosmob.ccc.test.Test$1) [duplicate 3]
15/11/05 15:04:04 ERROR scheduler.TaskSetManager: Task 0 in stage 4.0 failed 4 times; aborting job
15/11/05 15:04:04 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 
15/11/05 15:04:04 INFO scheduler.TaskSchedulerImpl: Cancelling stage 4
15/11/05 15:04:04 INFO scheduler.DAGScheduler: Failed to run take at DStream.scala:608
15/11/05 15:04:04 ERROR scheduler.JobScheduler: Error running job streaming job 1446725044000 ms.0
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 4 times, most recent failure: Lost task 0.3 in stage 4.0 (TID 78, qmic-tag-s05.quwic.org): java.lang.ClassNotFoundException: org.cosmob.ccc.test.Test$1
        java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        java.security.AccessController.doPrivileged(Native Method)
        java.net.URLClassLoader.findClass(URLClassLoader.java:360)
        java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        java.lang.Class.forName0(Native Method)
        java.lang.Class.forName(Class.java:344)
        org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59)
        java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
        java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
        org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
        org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
        org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
        org.apache.spark.scheduler.Task.run(Task.scala:54)
        org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
    at akka.actor.ActorCell.invoke(ActorCell.scala:456)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
    at akka.dispatch.Mailbox.run(Mailbox.scala:219)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
15/11/05 15:04:05 INFO scheduler.ReceiverTracker: Stream 0 received 1 blocks
15/11/05 15:04:05 INFO scheduler.JobScheduler: Added jobs for time 1446725045000 ms
15/11/05 15:04:05 INFO scheduler.JobScheduler: Starting job streaming job 1446725045000 ms.0 from job set of time 1446725045000 ms
15/11/05 15:04:05 INFO spark.SparkContext: Starting job: take at DStream.scala:608
15/11/05 15:04:05 INFO scheduler.DAGScheduler: Got job 4 (take at DStream.scala:608) with 1 output partitions (allowLocal=true)
15/11/05 15:04:05 INFO scheduler.DAGScheduler: Final stage: Stage 5(take at DStream.scala:608)
15/11/05 15:04:05 INFO scheduler.DAGScheduler: Parents of final stage: List()
15/11/05 15:04:05 INFO scheduler.DAGScheduler: Missing parents: List()
15/11/05 15:04:05 INFO scheduler.DAGScheduler: Submitting Stage 5 (MappedRDD[31] at map at MappedDStream.scala:35), which has no missing parents
15/11/05 15:04:05 INFO storage.MemoryStore: ensureFreeSpace(1872) called with curMem=16422, maxMem=996566630
15/11/05 15:04:05 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 1872.0 B, free 950.4 MB)
15/11/05 15:04:05 INFO storage.MemoryStore: ensureFreeSpace(1260) called with curMem=18294, maxMem=996566630
15/11/05 15:04:05 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 1260.0 B, free 950.4 MB)
15/11/05 15:04:05 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on wassimd.QUWIC.LOCAL:62993 (size: 1260.0 B, free: 950.4 MB)
15/11/05 15:04:05 INFO storage.BlockManagerMaster: Updated info of block broadcast_5_piece0
15/11/05 15:04:05 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 5 (MappedRDD[31] at map at MappedDStream.scala:35)
15/11/05 15:04:05 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
15/11/05 15:04:05 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 79, qmic-tag-s05.quwic.org, NODE_LOCAL, 1071 bytes)
15/11/05 15:04:05 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on qmic-tag-s05.quwic.org:53916 (size: 1260.0 B, free: 258.8 MB)
15/11/05 15:04:05 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 5.0 (TID 79, qmic-tag-s05.quwic.org): java.lang.ClassNotFoundException: org.cosmob.ccc.test.Test$1
        java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        java.security.AccessController.doPrivileged(Native Method)
        java.net.URLClassLoader.findClass(URLClassLoader.java:360)
        java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        java.lang.Class.forName0(Native Method)
        java.lang.Class.forName(Class.java:344)
        org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59)
        java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
        java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
        org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
        org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
        org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
        org.apache.spark.scheduler.Task.run(Task.scala:54)
        org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        java.lang.Thread.run(Thread.java:745)
15/11/05 15:04:05 INFO scheduler.TaskSetManager: Starting task 0.1 in stage 5.0 (TID 80, qmic-tag-s05.quwic.org, NODE_LOCAL, 1071 bytes)
15/11/05 15:04:05 INFO scheduler.TaskSetManager: Lost task 0.1 in stage 5.0 (TID 80) on executor qmic-tag-s05.quwic.org: java.lang.ClassNotFoundException (org.cosmob.ccc.test.Test$1) [duplicate 1]
15/11/05 15:04:05 INFO scheduler.TaskSetManager: Starting task 0.2 in stage 5.0 (TID 81, qmic-tag-s05.quwic.org, NODE_LOCAL, 1071 bytes)
15/11/05 15:04:05 INFO scheduler.TaskSetManager: Lost task 0.2 in stage 5.0 (TID 81) on executor qmic-tag-s05.quwic.org: java.lang.ClassNotFoundException (org.cosmob.ccc.test.Test$1) [duplicate 2]
15/11/05 15:04:05 INFO scheduler.TaskSetManager: Starting task 0.3 in stage 5.0 (TID 82, qmic-tag-s05.quwic.org, NODE_LOCAL, 1071 bytes)
15/11/05 15:04:05 INFO scheduler.TaskSetManager: Lost task 0.3 in stage 5.0 (TID 82) on executor qmic-tag-s05.quwic.org: java.lang.ClassNotFoundException (org.cosmob.ccc.test.Test$1) [duplicate 3]
15/11/05 15:04:05 ERROR scheduler.TaskSetManager: Task 0 in stage 5.0 failed 4 times; aborting job
15/11/05 15:04:05 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 
15/11/05 15:04:05 INFO scheduler.TaskSchedulerImpl: Cancelling stage 5
15/11/05 15:04:05 INFO scheduler.DAGScheduler: Failed to run take at DStream.scala:608
15/11/05 15:04:05 ERROR scheduler.JobScheduler: Error running job streaming job 1446725045000 ms.0
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 5.0 failed 4 times, most recent failure: Lost task 0.3 in stage 5.0 (TID 82, qmic-tag-s05.quwic.org): java.lang.ClassNotFoundException: org.cosmob.ccc.test.Test$1
        java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        java.security.AccessController.doPrivileged(Native Method)
        java.net.URLClassLoader.findClass(URLClassLoader.java:360)
        java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        java.lang.Class.forName0(Native Method)
        java.lang.Class.forName(Class.java:344)
        org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59)
        java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
        java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
        org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
        org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
        org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
        org.apache.spark.scheduler.Task.run(Task.scala:54)
        org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
    at akka.actor.ActorCell.invoke(ActorCell.scala:456)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
    at akka.dispatch.Mailbox.run(Mailbox.scala:219)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

最佳答案

您需要在uber-jar中包含您的传递依赖项(此处为org.cosmob.ccc.spark)在将其传递给 Spark-Submit 之前。

您可以找到有关此错误的更多信息 here 。如果你使用 sbt 你可以用 sbt-assembly 解决这个问题，如果您使用 Maven，请考虑 maven-shade .

如果您使用 Eclipse，则答案相同:Eclipse 必须以相同的方式构建 uber-jar。好像话题已经是addressed on SO .

关于java - 未找到类 Spark Streaming 和 kafka，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/33544251/

文章推荐： java - HashMap 返回 NULL

文章推荐： java - 其余 CXF 中是否有生成和消耗注释的默认类型

apache-kafka - 由于 Windows 上的致命异常 (kafka.Kafka$)，错误退出 Kafka - Apache Kafka
我在 Windows 机器上启动 Kafka-Server 时出现以下错误。我已经从以下链接下载了 Scala 2.11 - kafka_2.11-2.1.0.tgz:https://kafka.ap
noclassdeffounderror - kafka NoClassDefFoundError kafka/Kafka
关于Apache-Kafka messaging queue . 我已经从 Kafka 下载页面下载了 Apache Kafka。我已将其提取到 /opt/apache/installed/kafka
apache-kafka - 当底层的 Kafka 主题被删除时，Kafka 服务器上的 Kafka 消费者组会发生什么？
假设我有 Kafka 主题 cars。我还有一个消费者组 cars-consumers 订阅了 cars 主题。 cars-consumers 消费者组当前位于偏移量 89。当我现在删除 cars
apache-kafka - 我应该用什么 : Kafka Stream or Kafka consumer api or Kafka connect
我想知道什么最适合我:Kafka 流或 Kafka 消费者 api 或 Kafka 连接？我想从主题中读取数据，然后进行一些处理并写入数据库。所以我编写了消费者，但我觉得我可以编写 Kafka 流应
apache-kafka - Kafka 流与 Kafka 消费者如何决定使用什么
我曾研究过一些 Kafka 流应用程序和 Kafka 消费者应用程序。最后，Kafka流不过是消费来自Kafka的实时事件的消费者。因此，我无法弄清楚何时使用 Kafka 流或为什么我们应该使用
apache-kafka - Kafka 确认与 Kafka 提交
Kafka Acknowledgement 和 Kafka 消费者 commitSync() 有什么区别两者都用于手动偏移管理，并希望两者同步工作。请协助最佳答案使用 spring-kafka
apache-kafka - 如何在 Kafka 代理上代理 Apache Kafka 生产者请求，并重定向到单独的 Kafka 集群？
如何在 Kafka 代理上代理 Apache Kafka 生产者请求，并重定向到单独的 Kafka 集群？在我的特定情况下，无法更新写入此集群的客户端。这意味着，执行以下操作是不可行的: 更新客户端
apache-kafka - ConsumerOffsetChecker 的 Kafka 10 kafka-consumer-groups.sh 与 Kafka 8 kafka-run-class.sh
我需要在 Kafka 10 中命名我的消费者，就像我在 Kafka 8 中所做的一样，因为我有脚本可以嗅出并进一步使用这些信息。显然，consumer.id 的默认命名已更改(并且现在还单独显示了
【kafka】kafka /log_dir_event_notification的LogDir脱机事件通知
1.概述我们会看到zk的数据中有一个节点/log_dir_event_notification/，这是一个序列号持久节点这个节点在kafka中承担的作用是: 当某个Broker上的LogDir出现
apache-kafka - 如何使用 kafka 工具集生成具有空值的 Kafka 记录
我正在使用以下命令: bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test.topic --property
spring-kafka - 如何针对服务器上运行的真实 kafka 代理测试 kafka 消费者？
我很难理解 Java Spring Boot 中的一些 Kafka 概念。我想针对在服务器上运行的真实 Kafka 代理测试消费者，该服务器有一些生产者已将数据写入/已经将数据写入各种主题。我想与服务
apache-kafka - 如何在 Kafka 流中获取当前的 Kafka 主题？
我的场景是我使用了很多共享前缀的 Kafka 主题(例如 house.door, house.room ) 并使用 Kafka 流正则表达式主题模式 API 使用所有主题。一切看起来都不错，我得到了
apache-kafka - kafka AdminClient 获取 kafka 版本
有没有办法以编程方式获取kafka集群的版本？例如，使用AdminClient应用程序接口(interface)。我想在消费者/生产者应用程序中识别 kafka 集群的版本。最佳答案目前无法检索
apache-kafka - KAFKA 重启问题 : Unable to restart kafka without deleting/tmp/kafka-logs
每当我尝试重新启动 kafka 时，它都会出现以下错误。一旦我删除/tmp/kafka-logs 它就会得到解决，但它也会删除我的主题。有办法解决吗？ ERROR Error while
apache-kafka - 从 Kafka 客户端增加 Kafka 主题中的分区数
我是 Apache Kafka 的新用户，我仍在了解内部结构。在我的用例中，我需要从 Kafka Producer 客户端动态增加主题的分区数。我发现了其他类似的 questions关于增加分区大
apache-kafka - 如何将主题从 kafka 集群复制到另一个 kafka 集群？
正如 Kafka 文档所示，一种方法是通过 kafka.tools.MirrorMaker 来实现这一点。但是，我需要将一个主题(比如测试带 1 个分区)(其内容和元数据)从生产环境复制到没有连接
apache-kafka - Spring kafka 和 Kafka 集群
我已经在集群中配置了 3 个 kafka，我正在尝试与 spring-kafka 一起使用。但是在我杀死 kafka 领导者之后，我无法将其他消息发送到队列中。我将 spring.kafka.bo
apache-kafka - 是否可以将偏移量重置为 kafka 连接器中的 kafka 消费者组的主题？
我的 kafka sink 连接器从多个主题(配置了 10 个任务)读取，并处理来自所有主题的 300 条记录。根据每个记录中保存的信息，连接器可以执行某些操作。以下是触发器记录中键值对的示例: "
apache-kafka - Kafka Streams 如何获取 kafka header
我有以下 kafka 流代码 public class KafkaStreamHandler implements Processor{ private ProcessorConte
apache-kafka - kafka 连接失败时的 kafka-streams 警报
当 kafka-streams 应用程序正在运行并且 Kafka 突然关闭时，应用程序进入“等待”模式，发送警告日志的消费者和生产者线程无法连接，当 Kafka 回来时，一切都应该(理论上)去恢复正常

行者123

个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章

滴滴打车优惠券免费领取

全站热门文章

首页

博学

6Ren·AI

商城

java - 未找到类 Spark Streaming 和 kafka