java - 在 Spark Web UI 中看不到完成的作业

Question

我正在使用 ./bin/spark-submit 来运行我的 Spark 作业。它运行良好，但打开 Spark Web UI，我在完成列表中看不到作业。

./bin/spark-submit --name "myapp" --master local --conf "spark.master=spark://fahad:7077" --class com.apptest.App ~/app-0.0.1-SNAPSHOT.jar

注意:Spark 版本 2.0.1，1 个 worker 正在运行，master UI 位于 localhost:8080 worker 和 master 都从 ./sbin/start-*.sh 运行脚本。

Answer 1

有两种不同的 UI，常规 Spark UI 和 Spark History Server。

在作业完成后显示作业的是历史服务器。

http://spark.apache.org/docs/latest/monitoring.html

他们在文档中解释说您需要通过运行来启动它:

./sbin/start-history-server.sh

This creates a web interface at http://server-url:18080 by default, listing incomplete and completed applications and attempts.

When using the file-system provider class (see spark.history.provider below), the base logging directory must be supplied in the spark.history.fs.logDirectory configuration option, and should contain sub-directories that each represents an application’s event logs.

The spark jobs themselves must be configured to log events, and to log them to the same shared, writeable directory. For example, if the server was configured with a log directory of hdfs://namenode/shared/spark-logs, then the client-side options would be:

spark.eventLog.enabled true spark.eventLog.dir hdfs://namenode/shared/spark-logs