apache-spark - 停止 Spark Streaming 中的默认输出

Question

有没有办法禁用 Spark Streaming 中显示的默认输出。下面给出的是运行 Spark 流应用程序时打印的默认输出。我只想看到我添加的日志消息(在驱动程序中)被打印

spark-submit --class a11_TCP_IP.a11_Basic.a11_HelloTcp target/scala-2.10/sparkstreamingexamples_2.10-1.0.jar

16/04/27 13:20:15 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data
16/04/27 13:20:15 ERROR ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Socket data stream had no more data
16/04/27 13:20:17 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data
16/04/27 13:20:17 ERROR ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Socket data stream had no more data
-------------------------------------------
Time: 1461777618000 ms
-------------------------------------------

16/04/27 13:20:19 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data
16/04/27 13:20:19 ERROR ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Socket data stream had no more data
-------------------------------------------
Time: 1461777621000 ms
-------------------------------------------

16/04/27 13:20:21 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data

Answer 1

您的 Spark 文件夹应该有一个 conf/ 目录。其中有一个 log4j.properties 文件。如果没有，请将 conf/log4j.properties.template 复制到 conf/log4j.properties。

现在在编辑器中打开 conf/log4j.properties，并替换此行:

log4j.rootCategory=INFO, console

有了这两个:

log4j.rootCategory=INFO, null
log4j.appender.null=org.apache.log4j.varia.NullAppender