gpt4 book ai didi

docker - Heapster 无法从 Kubernetes 集群上的 Kubelet 获取容器统计信息

转载 作者:行者123 更新时间:2023-12-02 11:41:41 30 4
gpt4 key购买 nike

我已经基于 Running Kubernetes Locally via Docker 在 Ubuntu (trusty) 上建立了一个 Kubernetes 集群。指导、部署 DNS 并使用 InfluxDB 后端和 Grafana UI 运行 Heapster。

除了 Grafana 之外,一切似乎都运行顺利,它不显示任何图表,但消息 No datapoints在其图表中:Screenshot

检查 Docker 容器日志后,我发现 Heapster 无法访问 kubelet API(?),因此没有任何指标被持久化到 InfluxDB 中:

user@host:~$ docker logs e490a3ac10a8
I0701 07:07:30.829745 1 heapster.go:65] /heapster --source=kubernetes:https://kubernetes.default --sink=influxdb:http://monitoring-influxdb:8086
I0701 07:07:30.830082 1 heapster.go:66] Heapster version 1.2.0-beta.0
I0701 07:07:30.830809 1 configs.go:60] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0701 07:07:30.831284 1 configs.go:61] Using kubelet port 10255
E0701 07:09:38.196674 1 influxdb.go:209] issues while creating an InfluxDB sink: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp 10.0.0.223:8086: getsockopt: connection timed out, will retry on use
I0701 07:09:38.196919 1 influxdb.go:223] created influxdb sink with options: host:monitoring-influxdb:8086 user:root db:k8s
I0701 07:09:38.197048 1 heapster.go:92] Starting with InfluxDB Sink
I0701 07:09:38.197154 1 heapster.go:92] Starting with Metric Sink
I0701 07:09:38.228046 1 heapster.go:171] Starting heapster on port 8082
I0701 07:10:05.000370 1 manager.go:79] Scraping metrics start: 2016-07-01 07:09:00 +0000 UTC, end: 2016-07-01 07:10:00 +0000 UTC
E0701 07:10:05.008785 1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
I0701 07:10:05.009119 1 manager.go:152] ScrapeMetrics: time: 8.013178ms size: 0
I0701 07:11:05.001185 1 manager.go:79] Scraping metrics start: 2016-07-01 07:10:00 +0000 UTC, end: 2016-07-01 07:11:00 +0000 UTC
E0701 07:11:05.007130 1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
I0701 07:11:05.007686 1 manager.go:152] ScrapeMetrics: time: 5.945236ms size: 0
W0701 07:11:25.010298 1 manager.go:119] Failed to push data to sink: InfluxDB Sink
I0701 07:12:05.000420 1 manager.go:79] Scraping metrics start: 2016-07-01 07:11:00 +0000 UTC, end: 2016-07-01 07:12:00 +0000 UTC
E0701 07:12:05.002413 1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
I0701 07:12:05.002467 1 manager.go:152] ScrapeMetrics: time: 1.93825ms size: 0
E0701 07:12:12.309151 1 influxdb.go:150] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp 10.0.0.223:8086: getsockopt: connection timed out
I0701 07:12:12.351348 1 influxdb.go:201] Created database "k8s" on influxDB server at "monitoring-influxdb:8086"
I0701 07:13:05.001052 1 manager.go:79] Scraping metrics start: 2016-07-01 07:12:00 +0000 UTC, end: 2016-07-01 07:13:00 +0000 UTC
E0701 07:13:05.015947 1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
...

我在 GitHub 上发现了一些描述类似问题的问题,这些问题让我明白 Heapster 不是访问 kubelet(通过节点的环回)而是访问本身(通过容器的环回)。但是,我无法重现他们的解决方案:

github.com/kubernetes/heapster/issues/1183

You should either use host networking for Heapster pod or configure your cluster in a way that the node has a regular name not 127.0.0.1. The current problem is that node name is resolved to Heapster localhost. Please reopen in case of more problems.



-@piosz
  • 如何为我的 Heapster pod 启用“主机网络”?
  • 如何配置我的集群/节点以使用常规名称而不是 127.0.0.1?

  • github.com/kubernetes/heapster/issues/744

    Fixed by using better options in hyperkube, thanks for the help!



    -@ddispaltro
  • 有没有办法通过在 docker run 中添加/修改 kubelet 的选项标志来解决这个问题? ?我尝试设置 --hostname-override=<host's eth0 IP>--address=127.0.0.1 (正如这个 GitHub 问题的最后一个答案所建议的那样)但是 Heapster 的容器日志随后指出:I0701 08:23:05.000566 1 manager.go:79] Scraping metrics start: 2016-07-01 08:22:00 +0000 UTC, end: 2016-07-01 08:23:00 +0000 UTC
    E0701 08:23:05.000962 1 kubelet.go:279] Node 127.0.0.1 is not ready
    E0701 08:23:05.003018 1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://<host's eth0 IP>:10255/stats/container/": Post http://<host's eth0 IP>/stats/container/: dial tcp <host's eth0 IP>:10255: getsockopt: connection refused

  • 命名空间问题

    这个问题可能是由于我在 default 中运行 Kubernetes API 造成的吗? kube-system 中的命名空间和 Heapster ?
    user@host:~$ kubectl get --all-namespaces pods
    NAMESPACE NAME READY STATUS RESTARTS AGE
    default k8s-etcd-127.0.0.1 1/1 Running 0 18h
    default k8s-master-127.0.0.1 4/4 Running 1 18h
    default k8s-proxy-127.0.0.1 1/1 Running 0 18h
    kube-system heapster-lizks 1/1 Running 0 18h
    kube-system influxdb-grafana-e0pk2 2/2 Running 0 18h
    kube-system kube-dns-v10-4vjhm 4/4 Running 0 18h

    操作系统:Ubuntu 14.04.4 LTS(值得信赖)|
    Kubernetes:v1.2.5 |
    docker :v1.11.2

    最佳答案

    Heapster 已从 Kubernetes 获得节点列表,现在正尝试从每个节点上的 kublete 进程中提取统计信息(该节点有一个内置的 cAdvisor 收集节点上的统计信息)。在这种情况下,只有一个节点,并且 127.0.0.1 为 kubernetes 所知。这就是问题所在。 Heapster 容器正在尝试访问 127.0.0.1 的节点,该节点本身就是它本身,当然在 Heapster 容器中找不到要询问的 kublete 进程。

    要解决这个问题,需要做两件事。

  • 我们需要通过环回网络地址 127.0.0.1
  • 以外的其他方式引用 kublete 工作节点(我们运行 kubernetes 的主机)。
  • kublete 进程需要接受来自新网络接口(interface)/地址
  • 的流量

    假设您正在使用本地安装指南并使用以下命令启动 kubernetes
    hack/local-up-cluster.sh

    更改引用 kublete 的主机名非常简单。您可以采取更精细的方法,但将其设置为您的 eth0 ip 对我来说效果很好(ifconfig eth0)。缺点是您需要一个 eth0 接口(interface),这受制于 DHCP,因此您的里程可能会因使用方便而有所不同。
    export HOSTNAME_OVERRIDE=10.0.2.15

    让 kublete 进程接受来自任何网络接口(interface)的流量同样简单。
    export KUBELET_HOST=0.0.0.0

    关于docker - Heapster 无法从 Kubernetes 集群上的 Kubelet 获取容器统计信息,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/38142503/

    30 4 0
    Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
    广告合作:1813099741@qq.com 6ren.com