nginx - 在普罗米修斯中看不到 nginx 入口指标-6ren

nginx - 在普罗米修斯中看不到 nginx 入口指标

转载作者：行者123 更新时间：2023-12-05 03:44:50

我正在尝试让 prometheus 监控我的 ingress nginx 部署工作。我可以在 prometheus ui 中查看默认指标，但与 nginx 无关。

我已经通过集群监控部署了 prometheus-operator(同时更新了 serviceMonitorSelector 和 serviceMonitorNamespaceSelector 值):https://github.com/carlosedp/cluster-monitoring
我已经通过官方 helm chart 部署了 ingress-nginx:https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx

kubectl get prometheus -A

...
      runAsUser: 1000
    serviceAccountName: prometheus-k8s
    serviceMonitorNamespaceSelector:
      matchLabels:
        release: prometheus-operator
    serviceMonitorSelector:
      matchLabels:
        release: prometheus-operator
    version: v2.19.1
...

kubectl 获取 ns nginx

apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2021-02-08T02:37:35Z"
  labels:
    release: prometheus-operator
  managedFields:
  - apiVersion: v1
...

kubectl get servicemonitor nginx-ingress-nginx-controller -n monitoring -o yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: nginx
    meta.helm.sh/release-namespace: nginx
  creationTimestamp: "2021-02-14T19:09:01Z"
  generation: 5
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    helm.sh/chart: ingress-nginx-3.23.0
    release: prometheus-operator
  managedFields:
...
spec:
  endpoints:
  - interval: 30s
    port: metrics
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      release: prometheus-operator

kubectl get svc nginx-ingress-nginx-controller-metrics -n nginx -o yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: nginx
    meta.helm.sh/release-namespace: nginx
    prometheus.io/port: "10254"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-02-11T03:58:38Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    helm.sh/chart: ingress-nginx-3.23.0
    release: prometheus-operator
...
spec:
  clusterIP: 10.43.90.125
  clusterIPs:
  - 10.43.90.125
  ports:
  - name: metrics
    port: 9913
    protocol: TCP
    targetPort: metrics
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/name: ingress-nginx
  sessionAffinity: None
  type: ClusterIP

更新 2/17:kubectl get prometheus -n monitoring -o yaml

的完整输出

➜  ~ kubectl get -n monitoring prometheus -o yaml
apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: Prometheus
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"monitoring.coreos.com/v1","kind":"Prometheus","metadata":{"annotations":{},"labels":{"prometheus":"k8s"},"name":"k8s","namespace":"monitoring"},"spec":{"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"labelSelector":{"matchExpressions":[{"key":"prometheus","operator":"In","values":["k8s"]}]},"namespaces":["monitoring"],"topologyKey":"kubernetes.io/hostname"},"weight":100}]}},"alerting":{"alertmanagers":[{"name":"alertmanager-main","namespace":"monitoring","port":"web"}]},"externalUrl":"http://prometheus.192.168.1.15.nip.io","image":"prom/prometheus:v2.19.1","nodeSelector":{"kubernetes.io/os":"linux"},"podMonitorNamespaceSelector":{},"podMonitorSelector":{},"replicas":1,"resources":{"requests":{"memory":"400Mi"}},"retention":"15d","ruleSelector":{"matchLabels":{"prometheus":"k8s","role":"alert-rules"}},"scrapeInterval":"30s","scrapeTimeout":"30s","securityContext":{"fsGroup":2000,"runAsNonRoot":true,"runAsUser":1000},"serviceAccountName":"prometheus-k8s","serviceMonitorNamespaceSelector":{},"serviceMonitorSelector":{"matchLabels":{"release":"prometheus-operator"}},"version":"v2.19.1"}}
    creationTimestamp: "2021-02-13T04:13:49Z"
    generation: 4
    labels:
      prometheus: k8s
    managedFields:
    - apiVersion: monitoring.coreos.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:prometheus: {}
        f:spec:
          .: {}
          f:affinity:
            .: {}
            f:podAntiAffinity:
              .: {}
              f:preferredDuringSchedulingIgnoredDuringExecution: {}
          f:alerting:
            .: {}
            f:alertmanagers: {}
          f:externalUrl: {}
          f:image: {}
          f:nodeSelector:
            .: {}
            f:kubernetes.io/os: {}
          f:podMonitorNamespaceSelector: {}
          f:podMonitorSelector: {}
          f:replicas: {}
          f:resources:
            .: {}
            f:requests:
              .: {}
              f:memory: {}
          f:retention: {}
          f:ruleSelector:
            .: {}
            f:matchLabels:
              .: {}
              f:prometheus: {}
              f:role: {}
          f:scrapeInterval: {}
          f:scrapeTimeout: {}
          f:securityContext:
            .: {}
            f:fsGroup: {}
            f:runAsNonRoot: {}
            f:runAsUser: {}
          f:serviceAccountName: {}
          f:serviceMonitorNamespaceSelector: {}
          f:serviceMonitorSelector:
            .: {}
            f:matchLabels:
              .: {}
              f:release: {}
          f:version: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: "2021-02-15T15:07:05Z"
    name: k8s
    namespace: monitoring
    resourceVersion: "1321004"
    uid: a9829d00-2765-416b-b068-2c1e57623832
  spec:
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: prometheus
                operator: In
                values:
                - k8s
            namespaces:
            - monitoring
            topologyKey: kubernetes.io/hostname
          weight: 100
    alerting:
      alertmanagers:
      - name: alertmanager-main
        namespace: monitoring
        port: web
    externalUrl: http://prometheus.192.168.1.15.nip.io
    image: prom/prometheus:v2.19.1
    nodeSelector:
      kubernetes.io/os: linux
    podMonitorNamespaceSelector: {}
    podMonitorSelector: {}
    replicas: 1
    resources:
      requests:
        memory: 400Mi
    retention: 15d
    ruleSelector:
      matchLabels:
        prometheus: k8s
        role: alert-rules
    scrapeInterval: 30s
    scrapeTimeout: 30s
    securityContext:
      fsGroup: 2000
      runAsNonRoot: true
      runAsUser: 1000
    serviceAccountName: prometheus-k8s
    serviceMonitorNamespaceSelector: {}
    serviceMonitorSelector:
      matchLabels:
        release: prometheus-operator
    version: v2.19.1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

我也尝试过让监控与其他部署一起工作，但无济于事(所以我显然做错了什么)，但这是我的普罗米修斯日志:

➜  ~ kubectl logs prometheus-k8s-0  -n monitoring -c prometheus --tail 25
level=error ts=2021-02-17T15:29:18.588Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:29:25.978Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"mqtt\""
level=error ts=2021-02-17T15:29:43.562Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:29:46.748Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:29:56.841Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"pihole\""
level=error ts=2021-02-17T15:30:05.943Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:30:14.735Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"mqtt\""
level=error ts=2021-02-17T15:30:18.489Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:30:28.171Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"pihole\""
level=error ts=2021-02-17T15:30:30.412Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:30:57.878Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:30:59.128Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"mqtt\""
level=error ts=2021-02-17T15:31:15.689Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:31:21.136Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:31:27.498Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"pihole\""
level=error ts=2021-02-17T15:31:36.739Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"mqtt\""
level=error ts=2021-02-17T15:31:54.742Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:32:03.270Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:32:13.738Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:32:20.258Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"mqtt\""
level=error ts=2021-02-17T15:32:24.315Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"pihole\""
level=error ts=2021-02-17T15:32:44.099Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:32:58.299Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:33:03.925Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:33:05.050Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"mqtt\""

最佳答案

经过大量研究，我遇到的问题涉及 ClusterRole 定义。 prometheus 日志表明我试图监控的许多命名空间存在权限问题，因此我最终不得不向我的 ClusterRole 添加权限以包含 pods 等资源， services 和 endpoints，并包含 list 和 watch 等动词:

 apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: prometheus-k8s
  rules:
  - apiGroups:
    - ""
    resources:
    - nodes/metrics
    - nodes
    - services
    - endpoints
    - pods
    verbs:
    - get
    - list
    - watch
  - nonResourceURLs:
    - /metrics
    verbs:
    - get

关于nginx - 在普罗米修斯中看不到 nginx 入口指标，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/66216133/

文章推荐： machine-learning - CountVectorizer 如何处理测试数据中的新词？

文章推荐： node.js - TypeORM AfterInsert() 无法在另一个表中添加 userId

文章推荐： python - 错误的 "symbol"币安 API

nginx - nginx:配置文件/etc/nginx/nginx.conf测试失败(在上游找不到主机)
我有一个无所事事的盒子，已经运行了一段时间，今天，由于某种原因，当我尝试重新启动nginx时，得到了以下提示。 nginx: [emerg] host not found in upstream "w
nginx - 使用命令 "nginx"、 "service start nginx"和 "systemctl nginx start"启动 nginx 有什么区别？
我注意到，当我使用 ubuntu 命令“nginx”启动 nginx 并执行 systemctl status nginx 时。它表明 systemctl 已禁用。此外，如果我首先使用命令 syste
nginx - 如何将 Nginx 配置转换为 nginx-ingress？
我的 nginx 配置如下: proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $re
nginx - 启用了哪个 nginx-config 文件，/etc/nginx/conf.d/default.conf 还是/etc/nginx/nginx.conf？
周围有两个配置文件，/etc/nginx/conf.d/default.conf和 /etc/nginx/nginx.conf，但是启用了哪一个呢？我运行的是 CentOS6.4 和 nginx/1.
nginx - "/"以外的路由无法运行 nginx
我的 Nginx 配置仅适用于根位置，所有其他位置都返回“Cannot GET {location}”，其中位置是域后地址的其余部分。这是我的/etc/nginx/sites-enabled/def
nginx - nginx 只接受子域名
我在 nginx 中为 node.js 服务器设置了反向代理。 server { listen 80; server_name sub.domain.tld; location
nginx - Nginx 配置中静态位置的多个位置
我的应用程序将在两个位置提供静态文件，一个是/my/path/project/static，另一个是/my/path/project/jsutils/static。我很难让网络服务器在两个目录中查找
nginx - nginx 上的虚拟主机无法按预期运行
我的域名注册商的 DNS 访问我的服务器并获取 nginx 默认页面，因此配置正确我复制了一个当前正在工作的 nginx 虚拟主机，更改了 server_name和 conf 文件的名称，仅此而已。
nginx - nginx centos中的PhpPgAdmin登录失败
这个问题在这里已经有了答案: Can't login in to phpPgAdmin (2 个回答) 3年前关闭。我在centos中遇到了phpPgAdmin登录的奇怪问题，我做了所有需要的事情
nginx - Nginx Lua在proxy_pass之后修改HTML响应
我要为PoC进行的操作是向来自动态后端服务器的网页添加href。使用“ subs_filter”可以很容易地添加href，但是我需要使用响应中嵌入的信息来构造href。是否可以使用LUA处理来自pr
nginx - 如何从两个代理中识别realip？ (nginx)
我有网站服务器，它有两个代理(鱿鱼，CF)，它们使用不同的 header 来获取真实的 ip。我猜 nginx 命令 set_real_ip_from ;real_ip_header X-Forwa
nginx - 我无法重新启动我的 nginx
在控制台显示如下: Job for nginx.service failed because the control process exited witherror code. See "syste
nginx - 页面暂时不可用 - NGINX
我有一个问题，我怀疑是 NGINX 问题。基本上，当我尝试登录到我创建的网站时，出现以下错误…… 您要查找的页面暂时不可用。请稍后再试。有没有人以前遇到过这个？最佳答案如果 NGINX 虚拟主机
nginx - nginx 中的位置和文档路径
这是我的 nginx 配置文件: server { listen 80; server_name localhost; location / {
nginx - nginx 如何选择配置顺序？
在我的/etc/nginx/nginx.conf 文件中，我有配置。作为:- user nginx; worker_processes 1; error_log /var/log/nginx/e
nginx - nginx 有软退出吗？
有谁知道nginx支持软退出吗？这意味着它会一直运行直到所有连接都消失或超时(超过特定时间间隔)并且在此期间也不允许新连接吗？例如: nginx stop nginx running (2 conn
nginx - Nginx 可以用作直接服务器返回负载平衡器吗？
有没有办法将 Nginx 配置为类似这样的直接服务器返回 (DSR) 负载平衡器: http://blog.haproxy.com/2011/07/29/layer-4-load-balancing-
nginx - 使用附加模块重新编译 Nginx
我通过 apt-get 安装了 Nginx不久前在 Debian 上，我有几个网站在上面。现在我需要安装一些额外的模块，因为我不想搞砸任何事情，所以我想在执行之前仔细检查我的过程。希望这也能帮助其他不
nginx - Nginx 是否有等效的页面速度？
我知道 Apache 的 pagespeed 模块可以使页面访问更快，所以，我想知道 Nginx 是否有等效的模块？提前致谢! 最佳答案 https://github.com/pagespeed/n
nginx - Nginx worker_rlimit_nofile
如何将worker_rlimit_nofile设置为一个更大的数字，它可以是或建议最大为多少？我正在尝试遵循以下建议：大多数人遇到的第二个最大限制是与您的操作系统有关。打开一个shell，su给

行者123

个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章

滴滴打车优惠券免费领取

全站热门文章

首页

博学

6Ren·AI

商城

nginx - 在普罗米修斯中看不到 nginx 入口指标