hadoop - 为 Hadoop2 配置 Prometheus JMX 导出器

Question

我正在尝试使用 Prometheus JMX 导出器从以下运行在 ec2 实例上的 Hadoop2 守护进程中抓取指标:

• hadoop 名称节点
• hadoop 数据节点
• yarn 资源管理器
• yarn 节点管理器

我正在尝试将 JMX 导出器作为 java 代理与所有四个守护进程一起运行。为此，我在 hadoop-env.sh 和 yarn-env.sh 中添加了 EXTRA_JAVA_OPTS :

export HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS -javaagent:/home/ec2-user/jmx_exporter/jmx_prometheus_javaagent-0.10.jar=9102:/home/ec2-user/jmx_exporter/prometheus_config.yml"
export HADOOP_DATANODE_OPTS="$HADOOP_DATANODE_OPTS -javaagent:/home/ec2-user/jmx_exporter/jmx_prometheus_javaagent-0.10.jar=9102:/home/ec2-user/jmx_exporter/prometheus_config.yml"
export YARN_RESOURCEMANAGER_OPTS="$YARN_RESOURCEMANAGER_OPTS -javaagent:/home/ec2-user/jmx_exporter/jmx_prometheus_javaagent-0.10.jar=9102:/home/ec2-user/jmx_exporter/prometheus_config.yml"
export YARN_NODEMANAGER_OPTS="$YARN_NODEMANAGER_OPTS -javaagent:/home/ec2-user/jmx_exporter/jmx_prometheus_javaagent-0.10.jar=9102:/home/ec2-user/jmx_exporter/prometheus_config.yml"

资源管理器指标 NumAllSources 的示例 prometheus_config.yml 如下:

rules:
 - pattern: Hadoop<service=ResourceManager, name=MetricsSystem, sub=Stats><>NumAllSources
   name: sources
   labels:
    app_id: "hadoop_rm"

当我使用新配置和 java_opts 重新启动 resourcemanager 或其他守护进程时，出现以下异常:

Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:382)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:397)
Caused by: java.lang.IllegalArgumentException: Collector already registered that provides name: jmx_scrape_duration_seconds
at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:54)
at io.prometheus.jmx.shaded.io.prometheus.client.Collector.register(Collector.java:128)

关于如何解决这个问题有什么建议吗？

Answer 1

虽然 @chanhou 的解决方案可行，但我想将我的编辑保留在 hadoop-env.sh 中，所以我选择了

if ! grep -q <<<"$HADOOP_NAMENODE_OPTS" jmx_prometheus_javaagent; then
        HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS -javaagent:/home/caesarli/platform/jmx_prometheus_javaagent-0.12.0.jar=11099:/home/caesarli/platform/hadoop-2.8.4/etc/hadoop/jmx-name.yaml"
fi

和 HADOOP_DATANODE_OPTS 类似。