apache-spark - 如何访问 Spark Web UI？

Question

我正在本地运行 4 个节点的 Spark 应用程序。当我运行我的应用程序时，它显示我的驱动程序具有此地址 10.0.2.15:

INFO Utils: Successfully started service 'SparkUI' on port 4040.
INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.0.2.15:4040

在运行结束时它显示:

INFO SparkUI: Stopped Spark web UI at http://10.0.2.15:4040
INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
INFO MemoryStore: MemoryStore cleared
INFO BlockManager: BlockManager stopped
INFO BlockManagerMaster: BlockManagerMaster stopped
INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
INFO SparkContext: Successfully stopped SparkContext

我尝试通过以下方式访问 Spark Web: 10.0.2.15:4040但该页面无法访问。
尝试使用以下地址也没有帮助:

 http://localhost:18080

使用 ping 10.0.2.15结果是:

Send a request 'Ping' 10.0.2.15 with 32 bytes of data

Waiting time exceeded

Waiting time exceeded

Waiting time exceeded

Waiting time exceeded

Ping statistics for 10.0.2.15: Packages: sent = 4, received = 0, lost = 4 (100% loss)

使用 netstat -a 检查端口 4040 的可用性验证哪些端口可用。结果如下:

   Active connexion:

    Active       local address        Remote address                      state

    TCP          127.0.0.1:4040      DESKTOP-FF4U.....:0                 Listening

PS.:知道我的代码运行成功。
可能是什么原因？

Answer 1

INFO Utils: Successfully started service 'SparkUI' on port 4040.
INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.0.2.15:4040

这就是 Spark 报告 Web UI(内部称为 SparkUI)绑定(bind)到端口 4040 的方式。 .

只要 Spark 应用程序启动并运行，您就可以通过 http://10.0.2.15:4040 访问 Web UI .

INFO SparkUI: Stopped Spark web UI at http://10.0.2.15:4040
...
INFO SparkContext: Successfully stopped SparkContext

这是 Spark 应用程序完成的时间(它是否正确完成并不重要)。从现在开始，Web UI(位于 http://10.0.2.15:4040 )将不再可用。

I tried to access the Spark Web by: 10.0.2.15:4040 but the page is inaccessible.

这是 Spark 应用程序的预期行为。完成后， 4040 (这是 Web UI 的默认端口)不再可用。

Trying with the below address also didn't helped: http://localhost:18080

18080是 Spark History Server 的默认端口。它是一个单独的进程，无论运行 Spark 应用程序的可用性如何，它可能可用也可能不可用。

Spark History Server 与 Spark 应用程序完全不同。引用 official Spark docs :

It is still possible to construct the UI of an application through Spark’s history server, provided that the application’s event logs exist. You can start the history server by executing:

./sbin/start-history-server.sh

This creates a web interface at http://:18080 by default, listing incomplete and completed applications and attempts.

如您所见，您必须自己启动 Spark History Server 才能拥有 18080可用的。

此外，您必须使用 spark.eventLog.enabled和 spark.eventLog.dir配置属性，以便能够在 Spark 应用程序完成执行后查看它们的日志。引用 Spark official docs :

The spark jobs themselves must be configured to log events, and to log them to the same shared, writable directory. For example, if the server was configured with a log directory of hdfs://namenode/shared/spark-logs, then the client-side options would be:

spark.eventLog.enabled true
spark.eventLog.dir hdfs://namenode/shared/spark-logs