kubernetes - cAdvisor Prometheus container_cpu_load_average_10s 有两个值

Question

我有一个运行在 digital ocean 上的 kubernetes 集群，我想监控它。在 <apiserver>/api/v1/nodes/<nodename>/proxy/metrics/cadvisor 上查询公开的 cAdvisor 指标时我得到了 container_cpu_load_average_10s 的以下结果

# HELP container_cpu_load_average_10s Value of container cpu load average over the last 10 seconds.
# TYPE container_cpu_load_average_10s gauge
container_cpu_load_average_10s{container="",id="/",image="",name="",namespace="",pod=""} 0 1579564900287
container_cpu_load_average_10s{container="",id="/docker/0da952be93af76ef4f89c82d39ffc994814386013b0313db0e376ba8c1ca52ec",image="gcr.io/google-containers/hyperkube:v1.16.2",name="kubelet",namespace="",pod=""} 0 1579564899268
container_cpu_load_average_10s{container="",id="/kubepods",image="",name="",namespace="",pod=""} 0 1579564900316
container_cpu_load_average_10s{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0 1579564903221
container_cpu_load_average_10s{container="",id="/kubepods/besteffort/pod05e648ab-0d69-46e7-97f5-53fa5547e631",image="",name="",namespace="default",pod="sh2-74cdb7f89b-7wmn2"} 0 1579564889468
container_cpu_load_average_10s{container="",id="/kubepods/besteffort/pod1d3d6f5c-8b8f-47df-87e1-e6796b6c8cac",image="",name="",namespace="kube-system",pod="kubelet-rubber-stamp-7f966c6779-9pj2x"} 0 1579564897907
container_cpu_load_average_10s{container="",id="/kubepods/besteffort/pod35f81ba8-c778-4771-8103-ca6a1f1df3b3",image="",name="",namespace="kube-system",pod="cilium-operator-d5cd7d758-jlc7g"} 0 1579564902427
container_cpu_load_average_10s{container="",id="/kubepods/besteffort/pod7c42ac9d-14e2-4773-9f6b-78745e065d98",image="",name="",namespace="default",pod="sh-68d446d656-pr6lw"} 0 1579564893074
container_cpu_load_average_10s{container="",id="/kubepods/besteffort/pod87c517f4-be8d-4eeb-b550-7edd7b6629c7",image="",name="",namespace="ingress",pod="haproxy-ingress-c5fc9f5d-zbmc7"} 0 1579564903152
container_cpu_load_average_10s{container="",id="/kubepods/besteffort/poda137a036-0931-4d38-a39e-24269eda4558",image="",name="",namespace="kube-system",pod="metrics-server-7cdf9b7694-9ngsb"} 0 1579564906312

指标值实际上是两个值，第一个始终为 0，然后是 1579564906312 区域内的某个值。

我是普罗米修斯的新手，我认为一个指标只能有一个值，但显然 cadvisor 公开了两个值。这是一个错误还是我还不知道普罗米修斯的东西？如果它不是一个错误我应该如何处理它，因为普罗米修斯浏览器只显示第一个值为 0 的值。

Answer 1

每个指标都有维度。所以，这似乎是相同的指标，但它只是名称。这些指标的区别在于它们的标签。

如果您仔细查看屏幕截图，您会发现这些是不同服务的不同 pod 的 CPU 负载。

编辑:
Prometheus 公开其度量的格式是

metric_name [
  "{" label_name "=" `"` label_value `"` { "," label_name "=" `"` label_value `"` } [ "," ] "}"
] value [ timestamp ]

这暗示最后一个数字是时间戳。

阅读更多 https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md