kubernetes - 普罗米修斯联邦警报管理器

Question

我们有几个运行应用程序的集群。我们希望建立一个中央监视集群，该集群可以使用Prometheus Federation从其余集群中抓取指标。

为此，我需要在每个集群中安装prometheus服务器，并通过联合身份在中央集群中安装prometheus服务器。我还将在中央集群中也安装Grafana，以可视化我们从其他prometheus服务器收集的指标。

所以问题是；

我应该在哪里设置警报管理器？仅对于中央群集还是每个群集都必须同时是警报管理器？

使用联合身份验证时的最佳做法是什么？

虽然我可以使用入口 Controller 来公开每个Prometheus服务器？在Prometheus服务器和k8s中的联盟之间提供通信的最佳实践是什么？

Answer 1

基于此blog

Where should I setup the Alert Manager? Only for Central Cluster or each cluster has to be also alert manager?

What is the best practice alerting while using Federation?

答案是在每个群集上执行此操作。

If the data you need to do alerting is moved from one Prometheus to another then you've added an additional point of failure. This is particularly risky when WAN links such as the internet are involved. As far as is possible, you should try and push alerting as deep down the federation hierarchy as possible. For example an alert about a target being down should be setup on the Prometheus scraping that target, not a global Prometheus which could be several steps removed.

I though ı can use ingress controller to expose each prometheus server? What is the best practice to provide communication between prometheus server and federation in k8s?

我认为这取决于用例，在我检查过的每个文档中，它们仅使用prometheus.yml中 scrape_configs.static_configs中的目标

像 here

scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s

    honor_labels: true
    metrics_path: '/federate'

    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'

    static_configs:
      - targets:
        - 'source-prometheus-1:9090'
        - 'source-prometheus-2:9090'
        - 'source-prometheus-3:9090'

或

像 here

prometheus.yml:
    rule_files:
      - /etc/config/rules
      - /etc/config/alerts

    scrape_configs:
      - job_name: 'federate'
        scrape_interval: 15s

        honor_labels: true
        metrics_path: '/federate'

        params:
          'match[]':
            - '{job="prometheus"}'
            - '{__name__=~"job:.*"}'

        static_configs:
          - targets:
            - 'prometheus-server:80'

此外，值得检查他们如何在 tutorial中做到这一点，他们使用 helm在两个集群上使用两个Prometheus服务器构建中央监视集群。