apache-spark - 集群部署模式下的 Spark-submit 如何管理应用程序 Jars

Question

在《Spark in Action》一书中，我正在阅读:

“If you’re submitting your application in cluster-deploy mode using the spark-submit script, the JAR file you specify needs to be available on the worker (at the location you specified) that will be executing the application. Because there’s no way to say in advance which worker will execute your driver, you should put your application’s JAR file on all the workers if you intend to use cluster-deploy mode, or you can put your application’s JAR file on HDFS and use the HDFS URL as the JAR filename.”

但在官方文档中，我看到了这一点:

1 - If your code depends on other projects, you will need to package them alongside your application in order to distribute the code to a Spark cluster. To do this, create an assembly jar (or “uber” jar) containing your code and its dependencies. Both sbt and Maven have assembly plugins. When creating assembly jars, list Spark and Hadoop as provided dependencies; these need not be bundled since they are provided by the cluster manager at runtime. Once you have an assembled jar you can call the bin/spark-submit script as shown here while passing your jar.

2-If your application is launched through Spark submit, then the application jar is automatically distributed to all worker nodes. For any additional jars that your application depends on, you should specify them through the --jars flag using comma as a delimiter (e.g. --jars jar1,jar2). To control the application’s configuration or execution environment, see Spark Configuration.

我在这里错过了什么？它是如何工作的 ？我是否需要在整个集群中部署我的程序集 jar(除了主节点)？

Answer 1

官方文档是正确的(正如我们所期望的)。
特尔；博士:
无需跨集群复制应用程序文件或依赖项即可使用 spark-submit 提交 Spark 作业。 . spark-submit负责将应用程序 jar 交付给执行程序。更重要的是，使用 --jars 指定的 jar 文件选项也由驱动程序上的文件服务器提供给所有执行程序，因此我们也不需要将任何依赖项复制到执行程序。 Spark 会为您处理这些。
更多详细信息请访问 Advanced Dependency Management页