apache-spark - 带有类型安全配置的 Spark 2.1.1

Question

我正在尝试使用 typesafeconfig 为我的 spark 应用程序支持一些外部配置文件。

我正在像这样(驱动程序)的应用程序代码中加载 application.conf 文件:

val config = ConfigFactory.load()
val myProp = config.getString("app.property")
val df = spark.read.avro(myProp)

application.conf 看起来像这样:

app.propety="some value"

spark-submit 执行看起来像这样:

spark-submit 
        --class com.myapp.Main \
        --conf spark.shuffle.service.enabled=true \
        --conf spark.dynamicAllocation.enabled=true \
        --conf spark.dynamicAllocation.minExecutors=56 \
        --conf spark.dynamicAllocation.maxExecutors=1000 \
        --driver-class-path $HOME/conf/*.conf \
        --files $HOME/conf/application.conf \
        my-app-0.0.1-SNAPSHOT.jar

似乎它不起作用，我得到:

Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'app'
    at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:124)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:147)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:159)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:164)
    at com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:206)
    at com.paypal.cfs.fpti.Main$.main(Main.scala:42)
    at com.paypal.cfs.fpti.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:750)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

查看日志，我确实看到“--files”有效，似乎是类路径问题...

18/03/13 01:08:30 INFO SparkContext: Added file file:/home/user/conf/application.conf at file:/home/user/conf/application.conf with timestamp 1520928510820
18/03/13 01:08:30 INFO Utils: Copying /home/user/conf/application.conf to /tmp/spark-2938fde1-fa4a-47af-8dc6-1c54b5e89d48/userFiles-c2cec57f-18c8-491d-8679-df7e7da45e05/application.conf

Answer 1

事实证明，我非常接近开始时的答案...以下是它对我有用的方法:

spark-submit \
    --class com.myapp.Main \
    --conf spark.shuffle.service.enabled=true \
    --conf spark.dynamicAllocation.enabled=true \
    --conf spark.dynamicAllocation.minExecutors=56 \
    --conf spark.dynamicAllocation.maxExecutors=1000 \
    --driver-class-path $APP_HOME/conf \
    --files $APP_HOME/conf/application.conf \
    $APP_HOME/my-app-0.0.1-SNAPSHOT.jar

然后 $APP_HOME 将包含以下内容:

conf/application.conf
my-app-0.0.1-SNAPSHOT.jar

我想您需要确保将 application.conf 放在文件夹中，这就是诀窍。