apache-spark - Spark配置优先级

Question

在代码中指定spark应用程序配置之间是否有任何区别或优先级:

SparkConf().setMaster(yarn)

并在命令行中指定它们

spark-submit --master yarn

Answer 1

是的，使用set()函数将最高优先级赋予用户代码中的配置。之后，标志通过spark-submit传递。

Properties set directly on the SparkConf take highest precedence, then flags passed to spark-submit or spark-shell, then options in the spark-defaults.conf file. A few configuration keys have been renamed since earlier versions of Spark; in such cases, the older key names are still accepted, but take lower precedence than any instance of the newer key.

Source