log4j - 如何抑制在 EMR 上运行的 spark-sql 的 INFO 消息？

Question

我正在 EMR 上运行 Spark，如 Run Spark and Spark SQL on Amazon Elastic MapReduce 中所述:

This tutorial walks you through installing and operating Spark, a fast and general engine for large-scale data processing, on an Amazon EMR cluster. You will also create and query a dataset in Amazon S3 using Spark SQL, and learn how to monitor Spark on an Amazon EMR cluster with Amazon CloudWatch.

我试图抑制 INFO通过编辑记录 $HOME/spark/conf/log4j.properties无济于事。

输出看起来像:

$ ./spark/bin/spark-sql
Spark assembly has been built with Hive, including Datanucleus jars on classpath
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/.versions/2.4.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/.versions/spark-1.1.1.e/lib/spark-assembly-1.1.1-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2014-12-14 20:59:01,819 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1009)) - mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
2014-12-14 20:59:01,825 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1009)) - mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
2014-12-14 20:59:01,825 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1009)) - mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
2014-12-14 20:59:01,825 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1009)) - mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack

如何抑制上面的 INFO 消息？

Answer 1

如果您知道要抑制新 EMR 集群的日志记录，您也可以在创建集群时添加配置选项。

EMR 接受 JSON 形式的配置选项，您可以直接将其输入 AWS 控制台，或在使用 CLI 时通过文件传入。

在这种情况下，为了将日志级别更改为 WARN ，这是 JSON:

[
  {
    "classification": "spark-log4j",
    "properties": {"log4j.rootCategory": "WARN, console"}
  }
]

在控制台中，您将在第一个创建步骤中添加以下内容:



或者，如果您使用 CLI 创建集群:

aws emr create-cluster <options here> --configurations config_file.json

您可以阅读更多 in the EMR documentation .