java - Spark : configuration file 'metrics.properties'

Question

Spark 的配置文件(用于检索指标)，即 /conf/metrics.properties，说明以下内容:

Within an instance, a "source" specifies a particular set of grouped metrics. there are two kinds of sources:

1. Spark internal sources, like MasterSource, WorkerSource, etc, which will collect a Spark component's internal state. Each instance is paired with a Spark source that is added automatically.

2. Common sources, like JvmSource, which will collect low level state. These can be added through configuration options and are then loaded using reflection.

下面是一些示例，如

master.source.jvm.class=org.apache.spark.metrics.source.JvmSource

但是，没有示例解释如何从 MasterSource 或 WorkerSource 等其他来源获取指标，并且仅替换 JvmSource 并不能解决问题。

建议？

Answer 1

MasterSource 或 WorkerSource(均为内部源)会通过指标自动跟踪，因此无需明确指定。

例如，

master.sink.cvs.class=org.apache.spark.metrics.sink.CsvSink

每 10 秒将主实例指标输出到 /tmp 目录。