docker - Kubernetes HPA为自定义指标获取了错误的当前值

Question

在实际度量标准值低于132500m / 500(根据Prometheus)的情况下，HPA状态显示100/500。

$ kubectl get hpa -n frontend --context testing
NAME       REFERENCE              TARGETS                               MINPODS   MAXPODS   REPLICAS   AGE
frontend   Deployment/streaming   50237440 / 629145600, 132500m / 500   2         5         2          4d

HPA list 是:

---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: frontend
  namespace: streaming
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: streaming
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metricName: redis_memory_used_rss_bytes
      targetAverageValue: 629145600
  - type: Pods
    pods:
      metricName: redis_db_keys
      targetAverageValue: 500

它应显示正常结果，例如:

$ kubectl get hpa -n streaming --context streaming-eu
NAME       REFERENCE              TARGETS                               MINPODS   MAXPODS   REPLICAS   AGE
frontend   Deployment/streaming   50237440 / 629145600, 87 / 500   2         5         2          4d

问题在于那个 132500m值是错误的(Prometheus查询报告一个正常值)。而且由于 HPA并未扩大该指标，因此我认为它的值(value)有些不同。

在HPA中使用 oliver006/redis_exporter及其指标作为自定义 Pod指标来重现此问题。

Kubernetes版本:

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c", GitTreeState:"clean", BuildDate:"2017-12-21T06:34:11Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}`
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.4-gke.1", GitCommit:"10e47a740d0036a4964280bd663c8500da58e3aa", GitTreeState:"clean", BuildDate:"2018-03-13T18:00:36Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

云提供商:

GKE 1.9.4

Answer 1

我认为这是一个指标转换问题。
这是有关问题的杰出贡献者comment，但它与http_requests指标有关:

if you look at the documentation for the Prometheus adapter, you'll see that all cumulative (counter) metrics are converted to rate metrics, since the HPA's algorithm in fundamentally incompatible with scaling on cumulative metrics directly (scaling on cumulative metrics directly doesn't make much sense in general).

In your case, your http_requests_total is being converted into http_requests, so it will always show up as milli-requests from the metrics API when using the Prometheus adapter.

因此，在您的情况下，它将返回类似于132500毫记录的内容。只需将值除以1000，您将获得正确的平均值。