c# - 确定 Kubernetes 中应用程序关闭的原因

Question

我有几个 .net Core 应用程序无故关闭。看起来这是自实现健康检查以来发生的，但我无法在 kubernetes 中看到终止命令。

cmd

kubectl describe pod mypod

输出 (重启次数这么高，因为晚上每天都关机；舞台环境)

Name:               mypod
...
Status:             Running
...
Controlled By:      ReplicaSet/mypod-deployment-6dbb6bcb65
Containers:
  myservice:
    State:          Running
      Started:      Fri, 01 Nov 2019 09:59:40 +0100
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 01 Nov 2019 07:19:07 +0100
      Finished:     Fri, 01 Nov 2019 09:59:37 +0100
    Ready:          True
    Restart Count:  19
    Liveness:       http-get http://:80/liveness delay=10s timeout=1s period=5s #success=1 #failure=10
    Readiness:      http-get http://:80/hc delay=10s timeout=1s period=5s #success=1 #failure=10
    ...
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
...
Events:
  Type     Reason     Age                    From                               Message
  ----     ------     ----                   ----                               -------
  Warning  Unhealthy  18m (x103 over 3h29m)  kubelet, aks-agentpool-40946522-0  Readiness probe failed: Get http://10.244.0.146:80/hc: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  18m (x29 over 122m)    kubelet, aks-agentpool-40946522-0  Liveness probe failed: Get http://10.244.0.146:80/liveness: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

这些是 pods 日志

cmd

kubectl logs mypod --previous

输出

Hosting environment: Production
Content root path: /app
Now listening on: http://[::]:80
Application started. Press Ctrl+C to shut down.
Application is shutting down...

来自 azure 的相应日志


cmd

kubectl get events

输出 (我在这里缺少的是killing-event。我的假设是pod没有重新启动，这是由多次失败的健康检查引起的)

LAST SEEN   TYPE      REASON                    OBJECT                                                      MESSAGE
39m         Normal    NodeHasSufficientDisk     node/aks-agentpool-40946522-0                               Node aks-agentpool-40946522-0 status is now: NodeHasSufficientDisk
39m         Normal    NodeHasSufficientMemory   node/aks-agentpool-40946522-0                               Node aks-agentpool-40946522-0 status is now: NodeHasSufficientMemory
39m         Normal    NodeHasNoDiskPressure     node/aks-agentpool-40946522-0                               Node aks-agentpool-40946522-0 status is now: NodeHasNoDiskPressure
39m         Normal    NodeReady                 node/aks-agentpool-40946522-0                               Node aks-agentpool-40946522-0 status is now: NodeReady
39m         Normal    CREATE                    ingress/my-ingress                                          Ingress default/ebizsuite-ingress
39m         Normal    CREATE                    ingress/my-ingress                                          Ingress default/ebizsuite-ingress
7m2s        Warning   Unhealthy                 pod/otherpod2                                               Readiness probe failed: Get http://10.244.0.158:80/hc: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
7m1s        Warning   Unhealthy                 pod/otherpod2                                               Liveness probe failed: Get http://10.244.0.158:80/liveness: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
40m         Warning   Unhealthy                 pod/otherpod2                                               Liveness probe failed: Get http://10.244.0.158:80/liveness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
44m         Warning   Unhealthy                 pod/otherpod1                                               Liveness probe failed: Get http://10.244.0.151:80/liveness: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
5m35s       Warning   Unhealthy                 pod/otherpod1                                               Readiness probe failed: Get http://10.244.0.151:80/hc: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
40m         Warning   Unhealthy                 pod/otherpod1                                               Readiness probe failed: Get http://10.244.0.151:80/hc: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
8m8s        Warning   Unhealthy                 pod/mypod                                                   Readiness probe failed: Get http://10.244.0.146:80/hc: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
8m7s        Warning   Unhealthy                 pod/mypod                                                   Liveness probe failed: Get http://10.244.0.146:80/liveness: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
0s          Warning   Unhealthy                 pod/otherpod1                                               Readiness probe failed: Get http://10.244.0.151:80/hc: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

从另一个 pod curl (我每秒都在一个很长的循环中执行此操作，并且除了 200 OK 之外从未收到过其他信息)

kubectl exec -t otherpod1 -- curl --fail http://10.244.0.146:80/hc

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
{"status":"Healthy","totalDuration":"00:00:00.0647250","entries":{"self":{"data":{},"duration":"00:00:00.0000012","status":"Healthy"},"warmup":{"data":{},"duration":"00:00:00.0000007","status":"Healthy"},"TimeDB-check":{"data":{},"duration":"00:00:00.0341533","status":"Healthy"},"time-blob-storage-check":{"data":{},"duration":"00:00:00.0108192","status":"Healthy"},"time-rabbitmqbus-check":{"data":{},"duration":"00:00:00.0646841","status":"Healthy"}}}100   454    0   454    0     0   6579      0 --:--:-- --:--:-- --:--:--  6579

curl

kubectl exec -t otherpod1 -- curl --fail http://10.244.0.146:80/liveness

Healthy  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     7    0     7    0     0   7000      0 --:--:-- --:--:-- --:--:--  7000

Answer 1

我认为你可以:

仅修改 livenessprobe 和 readinessprobe 检查 http://80 , 剪切 URL

中的路径

删除 livenessprobe 和 readinessprobe (enabled=false)

只需将延迟时间增加到 5 或 10 分钟，之后您就可以 kubectl exec -it <pod-name> sh/bash进入那个 pod 并进行调试。您可以使用命令 netstat检查您要在端口 80 上启动的服务与否。最后一件事，你可以用 readinessprobe 或 livenessprobe curl -v http://localhost 做同样的事情。 , 如果此命令返回不同的代码 200，这就是为什么您的 Pod 总是重新启动的原因。

希望这对你有帮助，伙计。