scala - 如何减少 Spark 运行时输出的冗长程度？

Question

如何减少 Spark 运行时生成的跟踪信息量？

默认值太冗长，

如何关闭它，并在需要时打开它。

谢谢

详细模式

scala> val la = sc.parallelize(List(12,4,5,3,4,4,6,781))
scala> la.collect
15/01/28 09:57:24 INFO SparkContext: Starting job: collect at <console>:15
15/01/28 09:57:24 INFO DAGScheduler: Got job 3 (collect at <console>:15) with 1 output 
...
15/01/28 09:57:24 INFO Executor: Running task 0.0 in stage 3.0 (TID 3)
15/01/28 09:57:24 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 626 bytes result sent to driver
15/01/28 09:57:24 INFO DAGScheduler: Stage 3 (collect at <console>:15) finished in 0.002 s
15/01/28 09:57:24 INFO DAGScheduler: Job 3 finished: collect at <console>:15, took 0.020061 s
res5: Array[Int] = Array(12, 4, 5, 3, 4, 4, 6, 781)

静音模式(预期)

scala> val la = sc.parallelize(List(12,4,5,3,4,4,6,781))
scala> la.collect
res5: Array[Int] = Array(12, 4, 5, 3, 4, 4, 6, 781)

Answer 1

Spark 1.4.1

sc.setLogLevel("WARN")

来自源代码中的注释:

Valid log levels include: ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN

Spark 2.x - 2.3.1

sparkSession.sparkContext().setLogLevel("WARN")

Spark 2.3.2

sparkSession.sparkContext.setLogLevel("WARN")