- android - 多次调用 OnPrimaryClipChangedListener
- android - 无法更新 RecyclerView 中的 TextView 字段
- android.database.CursorIndexOutOfBoundsException : Index 0 requested, 光标大小为 0
- android - 使用 AppCompat 时,我们是否需要明确指定其 UI 组件(Spinner、EditText)颜色
我是 Apache Spark 新手。
我正在 UDP 端口 8060 接收数据,我想捕获它并实时执行一些操作,为此我使用 Spark Streaming。这是我的主要类(class):
import java.util.Arrays;
import java.util.Comparator;
import java.util.Map;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaReceiverInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
public class Main {
public static void main(String[] args) throws Exception {
SparkConf conf = new SparkConf().setMaster("local").setAppName("NetworkWordCount");
JavaStreamingContext jssc = new JavaStreamingContext(conf, new Duration(1000));
JavaReceiverInputDStream<String> lines = jssc.receiverStream(new CustomReceiver("localhost",8060));
JavaDStream<String> hash = lines.flatMap(
new FlatMapFunction<String, String>() {
@Override public Iterable<String> call(String x) {
// System.out.println(x);
String[] s=x.split("\\}\\{");
return Arrays.asList(s);
}
});
hash.print();
jssc.start();
}
}
和 CustomReceiver 类:
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.ConnectException;
import java.net.DatagramPacket;
import java.net.DatagramSocket;
import org.apache.spark.storage.StorageLevel;
import org.apache.spark.streaming.receiver.Receiver;
public class CustomReceiver extends Receiver<String> {
String host = null;
int port = -1;
public CustomReceiver(String host_ , int port_) {
super(StorageLevel.MEMORY_AND_DISK_2());
host = host_;
port = port_;
}
public void onStart() {
// Start the thread that receives data over a connection
new Thread() {
@Override public void run() {
receive();
}
}.start();
}
public void onStop() {
// There is nothing much to do as the thread calling receive()
// is designed to stop by itself isStopped() returns false
}
/** Create a socket connection and receive data until receiver is stopped */
private void receive() {
DatagramSocket socket = null;
String userInput = null;
try {
// connect to the server
socket = new DatagramSocket(port);
byte[] receiveData = new byte[1024];
DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length);
socket.receive(receivePacket);
userInput = new String( receivePacket.getData());
// System.out.println(userInput);
// Until stopped or connection broken continue reading
// System.out.println(userInput);
store(userInput);
socket.close();
// Restart in an attempt to connect again when server is active again
restart("Trying to connect again");
} catch(ConnectException ce) {
// restart if could not connect to server
restart("Could not connect", ce);
} catch(Throwable t) {
// restart if there is any other error
restart("Error receiving data", t);
}
}
}
代码循环,但一旦我发送数据,我就会收到一个奇怪的错误,并且它停止工作:
15/01/20 17:44:53 INFO spark.SecurityManager: Changing view acls to: gautambajaj
15/01/20 17:44:53 INFO spark.SecurityManager: Changing modify acls to: gautambajaj
15/01/20 17:44:53 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(gautambajaj); users with modify permissions: Set(gautambajaj)
15/01/20 17:44:54 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/01/20 17:44:55 INFO Remoting: Starting remoting
15/01/20 17:44:55 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.34.71.121:63919]
15/01/20 17:44:55 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@10.34.71.121:63919]
15/01/20 17:44:55 INFO util.Utils: Successfully started service 'sparkDriver' on port 63919.
15/01/20 17:44:55 INFO spark.SparkEnv: Registering MapOutputTracker
15/01/20 17:44:55 INFO spark.SparkEnv: Registering BlockManagerMaster
15/01/20 17:44:55 INFO storage.DiskBlockManager: Created local directory at /var/folders/z3/347_jn996097tyhzl905tm7m0000gn/T/spark-local-20150120174455-e4cf
15/01/20 17:44:56 INFO util.Utils: Successfully started service 'Connection manager for block manager' on port 63920.
15/01/20 17:44:56 INFO network.ConnectionManager: Bound socket to port 63920 with id = ConnectionManagerId(10.34.71.121,63920)
15/01/20 17:44:56 INFO storage.MemoryStore: MemoryStore started with capacity 818.5 MB
15/01/20 17:44:56 INFO storage.BlockManagerMaster: Trying to register BlockManager
15/01/20 17:44:56 INFO storage.BlockManagerMasterActor: Registering block manager 10.34.71.121:63920 with 818.5 MB RAM, BlockManagerId(<driver>, 10.34.71.121, 63920, 0)
15/01/20 17:44:56 INFO storage.BlockManagerMaster: Registered BlockManager
15/01/20 17:44:56 INFO spark.HttpFileServer: HTTP File server directory is /var/folders/z3/347_jn996097tyhzl905tm7m0000gn/T/spark-9737478e-492e-44c6-accc-e8a7c91b2ab3
15/01/20 17:44:56 INFO spark.HttpServer: Starting HTTP Server
15/01/20 17:44:56 INFO server.Server: jetty-8.1.14.v20131031
15/01/20 17:44:56 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:63921
15/01/20 17:44:56 INFO util.Utils: Successfully started service 'HTTP file server' on port 63921.
15/01/20 17:44:57 INFO server.Server: jetty-8.1.14.v20131031
15/01/20 17:44:57 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
15/01/20 17:44:57 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
15/01/20 17:44:57 INFO ui.SparkUI: Started SparkUI at http://10.34.71.121:4040
2015-01-20 17:44:57.460 java[17010:1404608] Unable to load realm info from SCDynamicStore
15/01/20 17:44:57 INFO util.AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@10.34.71.121:63919/user/HeartbeatReceiver
15/01/20 17:44:58 INFO scheduler.ReceiverTracker: ReceiverTracker started
15/01/20 17:44:58 INFO dstream.ForEachDStream: metadataCleanupDelay = -1
15/01/20 17:44:58 INFO dstream.FlatMappedDStream: metadataCleanupDelay = -1
15/01/20 17:44:58 INFO dstream.PluggableInputDStream: metadataCleanupDelay = -1
15/01/20 17:44:58 INFO dstream.PluggableInputDStream: Slide time = 1000 ms
15/01/20 17:44:58 INFO dstream.PluggableInputDStream: Storage level = StorageLevel(false, false, false, false, 1)
15/01/20 17:44:58 INFO dstream.PluggableInputDStream: Checkpoint interval = null
15/01/20 17:44:58 INFO dstream.PluggableInputDStream: Remember duration = 1000 ms
15/01/20 17:44:58 INFO dstream.PluggableInputDStream: Initialized and validated org.apache.spark.streaming.dstream.PluggableInputDStream@e039859
15/01/20 17:44:58 INFO dstream.FlatMappedDStream: Slide time = 1000 ms
15/01/20 17:44:58 INFO dstream.FlatMappedDStream: Storage level = StorageLevel(false, false, false, false, 1)
15/01/20 17:44:58 INFO dstream.FlatMappedDStream: Checkpoint interval = null
15/01/20 17:44:58 INFO dstream.FlatMappedDStream: Remember duration = 1000 ms
15/01/20 17:44:58 INFO dstream.FlatMappedDStream: Initialized and validated org.apache.spark.streaming.dstream.FlatMappedDStream@6e247d4a
15/01/20 17:44:58 INFO dstream.ForEachDStream: Slide time = 1000 ms
15/01/20 17:44:58 INFO dstream.ForEachDStream: Storage level = StorageLevel(false, false, false, false, 1)
15/01/20 17:44:58 INFO dstream.ForEachDStream: Checkpoint interval = null
15/01/20 17:44:58 INFO dstream.ForEachDStream: Remember duration = 1000 ms
15/01/20 17:44:58 INFO dstream.ForEachDStream: Initialized and validated org.apache.spark.streaming.dstream.ForEachDStream@5f159e0c
15/01/20 17:44:58 INFO scheduler.ReceiverTracker: Starting 1 receivers
15/01/20 17:44:58 INFO spark.SparkContext: Starting job: start at Main.java:76
15/01/20 17:44:58 INFO scheduler.DAGScheduler: Got job 0 (start at Main.java:76) with 1 output partitions (allowLocal=false)
15/01/20 17:44:58 INFO scheduler.DAGScheduler: Final stage: Stage 0(start at Main.java:76)
15/01/20 17:44:58 INFO scheduler.DAGScheduler: Parents of final stage: List()
15/01/20 17:44:58 INFO util.RecurringTimer: Started timer for JobGenerator at time 1421743499000
15/01/20 17:44:58 INFO scheduler.JobGenerator: Started JobGenerator at 1421743499000 ms
15/01/20 17:44:58 INFO scheduler.JobScheduler: Started JobScheduler
15/01/20 17:44:58 INFO scheduler.DAGScheduler: Missing parents: List()
15/01/20 17:44:58 INFO scheduler.DAGScheduler: Submitting Stage 0 (ParallelCollectionRDD[0] at start at Main.java:76), which has no missing parents
15/01/20 17:44:59 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:44:59 INFO scheduler.JobScheduler: Added jobs for time 1421743499000 ms
15/01/20 17:44:59 INFO scheduler.JobScheduler: Starting job streaming job 1421743499000 ms.0 from job set of time 1421743499000 ms
-------------------------------------------
Time: 1421743499000 ms
-------------------------------------------
15/01/20 17:44:59 INFO scheduler.JobScheduler: Finished job streaming job 1421743499000 ms.0 from job set of time 1421743499000 ms
15/01/20 17:44:59 INFO scheduler.JobScheduler: Total delay: 0.302 s for time 1421743499000 ms (execution: 0.069 s)
15/01/20 17:45:00 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:00 INFO scheduler.JobScheduler: Added jobs for time 1421743500000 ms
15/01/20 17:45:00 INFO scheduler.JobScheduler: Starting job streaming job 1421743500000 ms.0 from job set of time 1421743500000 ms
-------------------------------------------
Time: 1421743500000 ms
-------------------------------------------
15/01/20 17:45:00 INFO scheduler.JobScheduler: Finished job streaming job 1421743500000 ms.0 from job set of time 1421743500000 ms
15/01/20 17:45:00 INFO scheduler.JobScheduler: Total delay: 0.005 s for time 1421743500000 ms (execution: 0.001 s)
15/01/20 17:45:00 INFO rdd.FlatMappedRDD: Removing RDD 2 from persistence list
15/01/20 17:45:00 INFO storage.BlockManager: Removing RDD 2
15/01/20 17:45:00 INFO rdd.BlockRDD: Removing RDD 1 from persistence list
15/01/20 17:45:00 INFO storage.BlockManager: Removing RDD 1
15/01/20 17:45:00 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[1] at receiverStream at Main.java:30 of time 1421743500000 ms
15/01/20 17:45:01 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:01 INFO scheduler.JobScheduler: Added jobs for time 1421743501000 ms
-------------------------------------------15/01/20 17:45:01 INFO scheduler.JobScheduler: Starting job streaming job 1421743501000 ms.0 from job set of time 1421743501000 ms
15/01/20 17:45:01 INFO scheduler.JobScheduler: Finished job streaming job 1421743501000 ms.0 from job set of time 1421743501000 ms
15/01/20 17:45:01 INFO rdd.FlatMappedRDD: Removing RDD 4 from persistence list
15/01/20 17:45:01 INFO scheduler.JobScheduler: Total delay: 0.004 s for time 1421743501000 ms (execution: 0.000 s)
Time: 1421743501000 ms
-------------------------------------------
15/01/20 17:45:01 INFO storage.BlockManager: Removing RDD 4
15/01/20 17:45:01 INFO rdd.BlockRDD: Removing RDD 3 from persistence list
15/01/20 17:45:01 INFO storage.BlockManager: Removing RDD 3
15/01/20 17:45:01 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[3] at receiverStream at Main.java:30 of time 1421743501000 ms
15/01/20 17:45:01 INFO storage.MemoryStore: ensureFreeSpace(1216) called with curMem=0, maxMem=858229309
15/01/20 17:45:01 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1216.0 B, free 818.5 MB)
15/01/20 17:45:01 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 0 (ParallelCollectionRDD[0] at start at Main.java:76)
15/01/20 17:45:01 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
15/01/20 17:45:01 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, PROCESS_LOCAL, 1558 bytes)
15/01/20 17:45:01 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
15/01/20 17:45:01 INFO receiver.ReceiverSupervisorImpl: Registered receiver 0
15/01/20 17:45:01 INFO util.RecurringTimer: Started timer for BlockGenerator at time 1421743502000
15/01/20 17:45:01 INFO receiver.BlockGenerator: Started block pushing thread
15/01/20 17:45:01 INFO receiver.BlockGenerator: Started BlockGenerator
15/01/20 17:45:01 INFO receiver.ReceiverSupervisorImpl: Starting receiver
15/01/20 17:45:01 INFO receiver.ReceiverSupervisorImpl: Called receiver onStart
15/01/20 17:45:01 INFO scheduler.ReceiverTracker: Registered receiver for stream 0 from akka://sparkDriver
15/01/20 17:45:01 INFO scheduler.ReceiverTracker: Registered receiver for stream 0 from akka://sparkDriver
15/01/20 17:45:02 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:02 INFO scheduler.JobScheduler: Added jobs for time 1421743502000 ms
-------------------------------------------
Time: 1421743502000 ms
-------------------------------------------
15/01/20 17:45:02 INFO scheduler.JobScheduler: Starting job streaming job 1421743502000 ms.0 from job set of time 1421743502000 ms
15/01/20 17:45:02 INFO scheduler.JobScheduler: Finished job streaming job 1421743502000 ms.0 from job set of time 1421743502000 ms
15/01/20 17:45:02 INFO rdd.FlatMappedRDD: Removing RDD 6 from persistence list
15/01/20 17:45:02 INFO scheduler.JobScheduler: Total delay: 0.004 s for time 1421743502000 ms (execution: 0.000 s)
15/01/20 17:45:02 INFO storage.BlockManager: Removing RDD 6
15/01/20 17:45:02 INFO rdd.BlockRDD: Removing RDD 5 from persistence list
15/01/20 17:45:02 INFO storage.BlockManager: Removing RDD 5
15/01/20 17:45:02 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[5] at receiverStream at Main.java:30 of time 1421743502000 ms
15/01/20 17:45:03 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:03 INFO scheduler.JobScheduler: Added jobs for time 1421743503000 ms
-------------------------------------------
15/01/20 17:45:03 INFO scheduler.JobScheduler: Starting job streaming job 1421743503000 ms.0 from job set of time 1421743503000 ms
15/01/20 17:45:03 INFO scheduler.JobScheduler: Finished job streaming job 1421743503000 ms.0 from job set of time 1421743503000 ms
15/01/20 17:45:03 INFO rdd.FlatMappedRDD: Removing RDD 8 from persistence list
15/01/20 17:45:03 INFO scheduler.JobScheduler: Total delay: 0.006 s for time 1421743503000 ms (execution: 0.000 s)
15/01/20 17:45:03 INFO storage.BlockManager: Removing RDD 8
15/01/20 17:45:03 INFO rdd.BlockRDD: Removing RDD 7 from persistence list
Time: 1421743503000 ms
-------------------------------------------
15/01/20 17:45:03 INFO storage.BlockManager: Removing RDD 7
15/01/20 17:45:03 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[7] at receiverStream at Main.java:30 of time 1421743503000 ms
15/01/20 17:45:04 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:04 INFO scheduler.JobScheduler: Added jobs for time 1421743504000 ms
15/01/20 17:45:04 INFO scheduler.JobScheduler: Starting job streaming job 1421743504000 ms.0 from job set of time 1421743504000 ms
-------------------------------------------
Time: 1421743504000 ms
-------------------------------------------
15/01/20 17:45:04 INFO scheduler.JobScheduler: Finished job streaming job 1421743504000 ms.0 from job set of time 1421743504000 ms
15/01/20 17:45:04 INFO rdd.FlatMappedRDD: Removing RDD 10 from persistence list
15/01/20 17:45:04 INFO scheduler.JobScheduler: Total delay: 0.007 s for time 1421743504000 ms (execution: 0.000 s)
15/01/20 17:45:04 INFO storage.BlockManager: Removing RDD 10
15/01/20 17:45:04 INFO rdd.BlockRDD: Removing RDD 9 from persistence list
15/01/20 17:45:04 INFO storage.BlockManager: Removing RDD 9
15/01/20 17:45:04 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[9] at receiverStream at Main.java:30 of time 1421743504000 ms
15/01/20 17:45:05 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:05 INFO scheduler.JobScheduler: Added jobs for time 1421743505000 ms
15/01/20 17:45:05 INFO scheduler.JobScheduler: Starting job streaming job 1421743505000 ms.0 from job set of time 1421743505000 ms
-------------------------------------------
Time: 1421743505000 ms
-------------------------------------------
15/01/20 17:45:05 INFO scheduler.JobScheduler: Finished job streaming job 1421743505000 ms.0 from job set of time 1421743505000 ms
15/01/20 17:45:05 INFO rdd.FlatMappedRDD: Removing RDD 12 from persistence list
15/01/20 17:45:05 INFO scheduler.JobScheduler: Total delay: 0.006 s for time 1421743505000 ms (execution: 0.000 s)
15/01/20 17:45:05 INFO storage.BlockManager: Removing RDD 12
15/01/20 17:45:05 INFO rdd.BlockRDD: Removing RDD 11 from persistence list
15/01/20 17:45:05 INFO storage.BlockManager: Removing RDD 11
15/01/20 17:45:05 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[11] at receiverStream at Main.java:30 of time 1421743505000 ms
15/01/20 17:45:06 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:06 INFO scheduler.JobScheduler: Added jobs for time 1421743506000 ms
15/01/20 17:45:06 INFO scheduler.JobScheduler: Starting job streaming job 1421743506000 ms.0 from job set of time 1421743506000 ms
-------------------------------------------
Time: 1421743506000 ms
-------------------------------------------
15/01/20 17:45:06 INFO scheduler.JobScheduler: Finished job streaming job 1421743506000 ms.0 from job set of time 1421743506000 ms
15/01/20 17:45:06 INFO rdd.FlatMappedRDD: Removing RDD 14 from persistence list
15/01/20 17:45:06 INFO scheduler.JobScheduler: Total delay: 0.004 s for time 1421743506000 ms (execution: 0.000 s)
15/01/20 17:45:06 INFO storage.BlockManager: Removing RDD 14
15/01/20 17:45:06 INFO rdd.BlockRDD: Removing RDD 13 from persistence list
15/01/20 17:45:06 INFO storage.BlockManager: Removing RDD 13
15/01/20 17:45:06 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[13] at receiverStream at Main.java:30 of time 1421743506000 ms
15/01/20 17:45:07 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:07 INFO scheduler.JobScheduler: Added jobs for time 1421743507000 ms
-------------------------------------------
15/01/20 17:45:07 INFO scheduler.JobScheduler: Starting job streaming job 1421743507000 ms.0 from job set of time 1421743507000 ms
Time: 1421743507000 ms
-------------------------------------------
15/01/20 17:45:07 INFO scheduler.JobScheduler: Finished job streaming job 1421743507000 ms.0 from job set of time 1421743507000 ms
15/01/20 17:45:07 INFO rdd.FlatMappedRDD: Removing RDD 16 from persistence list
15/01/20 17:45:07 INFO scheduler.JobScheduler: Total delay: 0.004 s for time 1421743507000 ms (execution: 0.001 s)
15/01/20 17:45:07 INFO storage.BlockManager: Removing RDD 16
15/01/20 17:45:07 INFO rdd.BlockRDD: Removing RDD 15 from persistence list
15/01/20 17:45:07 INFO storage.BlockManager: Removing RDD 15
15/01/20 17:45:07 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[15] at receiverStream at Main.java:30 of time 1421743507000 ms
15/01/20 17:45:08 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:08 INFO scheduler.JobScheduler: Added jobs for time 1421743508000 ms
15/01/20 17:45:08 INFO scheduler.JobScheduler: Starting job streaming job 1421743508000 ms.0 from job set of time 1421743508000 ms
-------------------------------------------
Time: 1421743508000 ms
-------------------------------------------
15/01/20 17:45:08 INFO scheduler.JobScheduler: Finished job streaming job 1421743508000 ms.0 from job set of time 1421743508000 ms
15/01/20 17:45:08 INFO scheduler.JobScheduler: Total delay: 0.004 s for time 1421743508000 ms (execution: 0.000 s)
15/01/20 17:45:08 INFO rdd.FlatMappedRDD: Removing RDD 18 from persistence list
15/01/20 17:45:08 INFO storage.BlockManager: Removing RDD 18
15/01/20 17:45:08 INFO rdd.BlockRDD: Removing RDD 17 from persistence list
15/01/20 17:45:08 INFO storage.BlockManager: Removing RDD 17
15/01/20 17:45:08 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[17] at receiverStream at Main.java:30 of time 1421743508000 ms
15/01/20 17:45:09 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:09 INFO scheduler.JobScheduler: Added jobs for time 1421743509000 ms
15/01/20 17:45:09 INFO scheduler.JobScheduler: Starting job streaming job 1421743509000 ms.0 from job set of time 1421743509000 ms
15/01/20 17:45:09 INFO scheduler.JobScheduler: Finished job streaming job 1421743509000 ms.0 from job set of time 1421743509000 ms
-------------------------------------------
Time: 1421743509000 ms
-------------------------------------------
15/01/20 17:45:09 INFO rdd.FlatMappedRDD: Removing RDD 20 from persistence list
15/01/20 17:45:09 INFO scheduler.JobScheduler: Total delay: 0.011 s for time 1421743509000 ms (execution: 0.000 s)
15/01/20 17:45:09 INFO storage.BlockManager: Removing RDD 20
15/01/20 17:45:09 INFO rdd.BlockRDD: Removing RDD 19 from persistence list
15/01/20 17:45:09 INFO storage.BlockManager: Removing RDD 19
15/01/20 17:45:09 INFO dstream.PluggableInputDStream: Removing blocks of RDD BlockRDD[19] at receiverStream at Main.java:30 of time 1421743509000 ms
15/01/20 17:45:09 WARN receiver.ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Trying to connect again
15/01/20 17:45:09 INFO receiver.ReceiverSupervisorImpl: Stopping receiver with message: Restarting receiver with delay 2000ms: Trying to connect again:
15/01/20 17:45:09 INFO receiver.ReceiverSupervisorImpl: Called receiver onStop
15/01/20 17:45:09 INFO receiver.ReceiverSupervisorImpl: Deregistering receiver 0
15/01/20 17:45:09 ERROR scheduler.ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Trying to connect again
15/01/20 17:45:09 INFO receiver.ReceiverSupervisorImpl: Stopped receiver 0
15/01/20 17:45:09 INFO storage.MemoryStore: ensureFreeSpace(2120) called with curMem=1216, maxMem=858229309
15/01/20 17:45:09 INFO storage.MemoryStore: Block input-0-1421743509200 stored as values in memory (estimated size 2.1 KB, free 818.5 MB)
15/01/20 17:45:09 INFO storage.BlockManagerInfo: Added input-0-1421743509200 in memory on 10.34.71.121:63920 (size: 2.1 KB, free: 818.5 MB)
15/01/20 17:45:09 INFO storage.BlockManagerMaster: Updated info of block input-0-1421743509200
15/01/20 17:45:09 WARN storage.BlockManager: Block input-0-1421743509200 replicated to only 0 peer(s) instead of 1 peers
15/01/20 17:45:09 INFO receiver.BlockGenerator: Pushed block input-0-1421743509200
15/01/20 17:45:10 INFO scheduler.ReceiverTracker: Stream 0 received 1 blocks
15/01/20 17:45:10 INFO scheduler.JobScheduler: Added jobs for time 1421743510000 ms
15/01/20 17:45:10 INFO scheduler.JobScheduler: Starting job streaming job 1421743510000 ms.0 from job set of time 1421743510000 ms
15/01/20 17:45:10 INFO spark.SparkContext: Starting job: getCallSite at DStream.scala:294
15/01/20 17:45:10 INFO scheduler.DAGScheduler: Got job 1 (getCallSite at DStream.scala:294) with 1 output partitions (allowLocal=true)
15/01/20 17:45:10 INFO scheduler.DAGScheduler: Final stage: Stage 1(getCallSite at DStream.scala:294)
15/01/20 17:45:10 INFO scheduler.DAGScheduler: Parents of final stage: List()
15/01/20 17:45:10 INFO scheduler.DAGScheduler: Missing parents: List()
15/01/20 17:45:10 INFO scheduler.DAGScheduler: Submitting Stage 1 (FlatMappedRDD[24] at flatMap at Main.java:32), which has no missing parents
15/01/20 17:45:10 INFO storage.MemoryStore: ensureFreeSpace(1800) called with curMem=3336, maxMem=858229309
15/01/20 17:45:10 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 1800.0 B, free 818.5 MB)
15/01/20 17:45:10 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 1 (FlatMappedRDD[24] at flatMap at Main.java:32)
15/01/20 17:45:10 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
15/01/20 17:45:11 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:11 INFO scheduler.JobScheduler: Added jobs for time 1421743511000 ms
15/01/20 17:45:11 INFO receiver.ReceiverSupervisorImpl: Starting receiver again
15/01/20 17:45:11 INFO receiver.ReceiverSupervisorImpl: Starting receiver
15/01/20 17:45:11 INFO receiver.ReceiverSupervisorImpl: Called receiver onStart
15/01/20 17:45:11 INFO scheduler.ReceiverTracker: Registered receiver for stream 0 from akka://sparkDriver
15/01/20 17:45:11 INFO receiver.ReceiverSupervisorImpl: Receiver started again
15/01/20 17:45:12 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:12 INFO scheduler.JobScheduler: Added jobs for time 1421743512000 ms
15/01/20 17:45:13 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:13 INFO scheduler.JobScheduler: Added jobs for time 1421743513000 ms
15/01/20 17:45:14 INFO scheduler.ReceiverTracker: Stream 0 received 0 blocks
15/01/20 17:45:14 INFO scheduler.JobScheduler: Added jobs for time 1421743514000 ms
我到底做错了什么?
最佳答案
socket.receive
函数上没有 while 循环(或其他重复构造)。它接收一个 1024 位数据包,存储它,然后关闭套接字。该接收器无法接收多个数据包。 Spark 尝试重新启动接收器两次,然后失败。
关于java - 使用 Apache Spark 进行 UDP,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/28042415/
在流处理方面,Apache Beam和Apache Kafka之间有什么区别? 我也试图掌握技术和程序上的差异。 请通过您的经验报告来帮助我理解。 最佳答案 Beam是一种API,它以一种统一的方式使
有点n00b的问题。 如果我使用 Apache Ignite 进行消息传递和事件处理,是否还需要使用 Kafka? 与 Ignite 相比,Kafka 基本上会给我哪些(如果有的话)额外功能? 提前致
Apache MetaModel 是一个数据访问框架,它为发现、探索和查询不同类型的数据源提供了一个通用接口(interface)。 Apache Drill 是一种无架构的 SQL 查询引擎,它通过
Tomcat是一个广泛使用的java web服务器,而Apache也是一个web服务器,它们在实际项目使用中有什么不同? 经过一些研究,我有了一个简单的想法,比如, Apache Tomcat Ja
既然简单地使用 Apache 就足以运行许多 Web 应用程序,那么人们何时以及为什么除了 Apache 之外还使用 Tomcat? 最佳答案 Apache Tomcat是一个网络服务器和 Java
我在某个 VPS( friend 的带 cPanel 的 apache 服务器)上有一个帐户,我在那里有一个 public_html 目录。我们有大约 5-6 个网站: /home/myusernam
我目前正在尝试将模块加载到 Apache,使用 cmake 构建。该模块称为 mod_mapcache。它已成功构建并正确安装在/usr/lib/apache2/modules directroy 中
我对 url 中的问号有疑问。 例如:我有 url test.com/controller/action/part_1%3Fpart_2 (其中 %3F 是 url 编码的问号),并使用此重写规则:R
在同一台机器上,Apache 在端口 80 上运行,Tomcat 在端口 8080 上运行。 Apache 包括 html;css;js;文件并调用 tomcat 服务。 基本上 exampledom
Apache 1 和 Apache 2 的分支有什么区别? 使用一种或另一种的优点和缺点? 似乎 Apache 2 的缺点之一是使用大量内存,但也许它处理请求的速度更快? 最有趣的是 Apache 作
实际上,我们正在使用 Apache 网络服务器来托管我们的 REST-API。 脚本是用 Lua 编写的,并使用 mod-lua 映射。 例如来自 httpd.conf 的实际片段: [...] Lu
我在 apache 上的 ubuntu 中有一个虚拟主机,这不是我的主要配置,我有另一个网页作为我的主要网页,所以我想使用虚拟主机在同一个 IP 上设置这个。 urologyexpert.mx 是我的
我使用 Apache camel 已经很长时间了,发现它是满足各种系统集成相关业务需求的绝佳解决方案。但是几年前我遇到了 Apache Nifi 解决方案。经过一番谷歌搜索后,我发现虽然 Nifi 可
由于两者都是一次处理事件的流框架,这两种技术/流框架之间的核心架构差异是什么? 此外,在哪些特定用例中,一个比另一个更合适? 最佳答案 正如您所提到的,两者都是实时内存计算的流式平台。但是当您仔细观察
apache 文件(如 httpd.conf 和虚拟主机)中使用的语言名称是什么,例如 # Ensure that Apache listens on port 80 Listen 80 D
作为我学习过程的一部分,我认为如果我扩展更多关于 apache 的知识会很好。我有几个问题,虽然我知道有些内容可能需要相当冗长的解释,但我希望您能提供一个概述,以便我知道去哪里寻找。 (最好引用 mo
关闭。这个问题是opinion-based .它目前不接受答案。 想改善这个问题吗?更新问题,以便可以通过 editing this post 用事实和引文回答问题. 4 个月前关闭。 Improve
就目前而言,这个问题不适合我们的问答形式。我们希望答案得到事实、引用或专业知识的支持,但这个问题可能会引起辩论、争论、投票或扩展讨论。如果您觉得这个问题可以改进并可能重新打开,visit the he
这个问题在这里已经有了答案: Difference Between Apache Kafka and Camel (Broker vs Integration) (4 个回答) 3年前关闭。 据我所知
我有 2 个使用相同规则的子域,如下所示: RewriteEngine On RewriteCond %{REQUEST_FILENAME} !-f RewriteCond
我是一名优秀的程序员,十分优秀!