apache-spark - Spark 流

Question

我在 Pyspark 中使用“批处理间隔”= 30 秒的 Spark Streaming

ssc = StreamingContext(sc, 30)

然后我想使用 window() 函数获取过去一小时的数据，并每隔 30 秒对这些数据进行切片。

kvs = KafkaUtils.createDirectStream(ssc, [topic], {"metadata.broker.list": brokers})

counts = kvs.map(lambda (k, v): json.loads(v))\
            .map(TransformInData).window(108000)

我有一个错误

16/02/18 10:23:01 INFO JobScheduler: Added jobs for time 1455790980000 ms
16/02/18 10:23:30 INFO PythonTransformedDStream: Slicing from 1455683040000 ms to 1455791010000 ms (aligned to 1455683040000 ms and 1455791010000 ms)
16/02/18 10:23:30 INFO PythonTransformedDStream: Time 1455790650000 ms is invalid as zeroTime is 1455790650000 ms and slideDuration is 30000 ms and difference is 0 ms
16/02/18 10:23:31 INFO JobScheduler: Added jobs for time 1455791010000 ms

我读过这个https://groups.google.com/forum/#!topic/spark-users/GQoxJHAAtX4但我不明白为什么它不起作用

Answer 1

我遇到了同样的问题。升级到 Spark 2.0.1 修复了它。