- html - 出于某种原因,IE8 对我的 Sass 文件中继承的 html5 CSS 不友好?
- JMeter 在响应断言中使用 span 标签的问题
- html - 在 :hover and :active? 上具有不同效果的 CSS 动画
- html - 相对于居中的 html 内容固定的 CSS 重复背景?
在 GKE 上,我尝试使用就绪探针/活性探针,并使用监控发布警报 https://cloud.google.com/monitoring/alerts/using-alerting-ui
作为测试,我创建了一个具有就绪探针/活性探针的 pod。正如我所料,每次探测检查都失败了。
apiVersion: v1
kind: Pod
metadata:
labels:
test: liveness
name: liveness-http
spec:
containers:
- name: liveness
image: k8s.gcr.io/liveness
args:
- /server
readinessProbe:
httpGet:
path: /healthz
port: 8080
httpHeaders:
- name: X-Custom-Header
value: Awesome
initialDelaySeconds: 0
periodSeconds: 10
timeoutSeconds: 10
successThreshold: 1
failureThreshold: 3
livenessProbe:
httpGet:
path: /healthz
port: 8080
httpHeaders:
- name: X-Custom-Header
value: Awesome
initialDelaySeconds: 20
periodSeconds: 60
timeoutSeconds: 30
successThreshold: 1
failureThreshold: 3
并检查 GCP 日志,首先显示了基于 periodSeconds 的两个错误日志。
2021-02-21 13:26:30.000 JSTReadiness probe failed: HTTP probe failed with statuscode: 500
2021-02-21 13:26:40.000 JSTReadiness probe failed: HTTP probe failed with statuscode: 500
2021-02-21 13:25:40.000 JSTLiveness probe failed: HTTP probe failed with statuscode: 500
2021-02-21 13:26:40.000 JSTLiveness probe failed: HTTP probe failed with statuscode: 500
$ kubectl get event
LAST SEEN TYPE REASON OBJECT MESSAGE
30m Normal Pulling pod/liveness-http Pulling image "k8s.gcr.io/liveness"
25m Warning Unhealthy pod/liveness-http Readiness probe failed: HTTP probe failed with statuscode: 500
20m Warning BackOff pod/liveness-http Back-off restarting failed container
20m Normal Scheduled pod/liveness-http Successfully assigned default/liveness-http to gke-cluster-default-pool-8bc9c75c-rfgc
17m Normal Pulling pod/liveness-http Pulling image "k8s.gcr.io/liveness"
17m Normal Pulled pod/liveness-http Successfully pulled image "k8s.gcr.io/liveness"
17m Normal Created pod/liveness-http Created container liveness
20m Normal Started pod/liveness-http Started container liveness
4m59s Warning Unhealthy pod/liveness-http Readiness probe failed: HTTP probe failed with statuscode: 500
17m Warning Unhealthy pod/liveness-http Liveness probe failed: HTTP probe failed with statuscode: 500
17m Normal Killing pod/liveness-http Container liveness failed liveness probe, will be restarted
最佳答案
背景
在 Configure Liveness, Readiness and Startup Probes 文档中,您可以找到信息:
The kubelet uses
liveness probes
to know when to restart a container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a container in such a state can help to make the application more available despite bugs.The kubelet uses
readiness probes
to know when a container is ready to start accepting traffic. A Pod is considered ready when all of its containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balancers.
GKE
master 由 google 管理,因此您不会使用
kubelet
找到
CLI
日志(您可能会尝试使用
Stackdriver
)。我已经在
Kubeadm
集群上对其进行了测试,并将
verbosity
级别设置为
8
。
$ kubectl get events
时,您只会收到最后一小时的事件(它可以在 Kubernetes 设置中更改 -
Kubeadm
但我认为它不能在
GKE
中更改,因为 master 由 google 管理。)
$ kubectl get events
LAST SEEN TYPE REASON OBJECT MESSAGE
37m Normal Starting node/kubeadm Starting kubelet.
...
33m Normal Scheduled pod/liveness-http Successfully assigned default/liveness-http to kubeadm
33m Normal Pulling pod/liveness-http Pulling image "k8s.gcr.io/liveness"
33m Normal Pulled pod/liveness-http Successfully pulled image "k8s.gcr.io/liveness" in 893.953679ms
33m Normal Created pod/liveness-http Created container liveness
33m Normal Started pod/liveness-http Started container liveness
3m12s Warning Unhealthy pod/liveness-http Readiness probe failed: HTTP probe failed with statuscode: 500
30m Warning Unhealthy pod/liveness-http Liveness probe failed: HTTP probe failed with statuscode: 500
8m17s Warning BackOff pod/liveness-http Back-off restarting failed container
在
~1 hour
之后再次执行相同的命令。
$ kubectl get events
LAST SEEN TYPE REASON OBJECT MESSAGE
33s Normal Pulling pod/liveness-http Pulling image "k8s.gcr.io/liveness"
5m40s Warning Unhealthy pod/liveness-http Readiness probe failed: HTTP probe failed with statuscode: 500
15m Warning BackOff pod/liveness-http Back-off restarting failed container
测试
Readiness Probe
检查每 10 秒执行一次,持续时间超过一小时。
Mar 09 14:48:34 kubeadm kubelet[3855]: I0309 14:48:34.222085 3855 prober.go:117] Readiness probe for "liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a):liveness" failed (failure): HTTP probe failed with statuscode: 500
Mar 09 14:48:44 kubeadm kubelet[3855]: I0309 14:48:44.221782 3855 prober.go:117] Readiness probe for "liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a):liveness" failed (failure): HTTP probe failed with statuscode: 500
Mar 09 14:48:54 kubeadm kubelet[3855]: I0309 14:48:54.221828 3855 prober.go:117] Readiness probe for "liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a):liveness" failed (failure): HTTP probe failed with statuscode: 500
...
Mar 09 15:01:34 kubeadm kubelet[3855]: I0309 15:01:34.222491 3855 prober.go:117] Readiness probe for "liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4
562a):liveness" failed (failure): HTTP probe failed with statuscode: 500
Mar 09 15:01:44 kubeadm kubelet[3855]: I0309 15:01:44.221877 3855 prober.go:117] Readiness probe for "liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a):liveness" failed (failure): HTTP probe failed with statuscode: 500
Mar 09 15:01:54 kubeadm kubelet[3855]: I0309 15:01:54.221976 3855 prober.go:117] Readiness probe for "liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a):liveness" failed (failure): HTTP probe failed with statuscode: 500
...
Mar 09 15:10:14 kubeadm kubelet[3855]: I0309 15:10:14.222163 3855 prober.go:117] Readiness probe for "liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a):liveness" failed (failure): HTTP probe failed with statuscode: 500
Mar 09 15:10:24 kubeadm kubelet[3855]: I0309 15:10:24.221744 3855 prober.go:117] Readiness probe for "liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a):liveness" failed (failure): HTTP probe failed with statuscode: 500
Mar 09 15:10:34 kubeadm kubelet[3855]: I0309 15:10:34.223877 3855 prober.go:117] Readiness probe for "liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a):liveness" failed (failure): HTTP probe failed with statuscode: 500
...
Mar 09 16:04:14 kubeadm kubelet[3855]: I0309 16:04:14.222853 3855 prober.go:117] Readiness probe for "liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a):liveness" failed (failure): HTTP probe failed with statuscode: 500
Mar 09 16:04:24 kubeadm kubelet[3855]: I0309 16:04:24.222531 3855 prober.go:117] Readiness probe for "liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a):liveness" failed (failure): HTTP probe failed with statuscode: 500
此外,还有
Liveness probe
条目。
Mar 09 16:12:58 kubeadm kubelet[3855]: I0309 16:12:58.462878 3855 prober.go:117] Liveness probe for "liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a):liveness" failed (failure): HTTP probe failed with statuscode: 500
Mar 09 16:13:58 kubeadm kubelet[3855]: I0309 16:13:58.462906 3855 prober.go:117] Liveness probe for "liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a):liveness" failed (failure): HTTP probe failed with statuscode: 500
Mar 09 16:14:58 kubeadm kubelet[3855]: I0309 16:14:58.465470 3855 kuberuntime_manager.go:656] Container "liveness" ({"docker" "95567f85708ffac8b34b6c6f2bdb4
9d8eb57e7704b7b416083c7f296dd40cd0b"}) of pod liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a): Container liveness failed liveness probe, will be restarted
Mar 09 16:14:58 kubeadm kubelet[3855]: I0309 16:14:58.465587 3855 kuberuntime_manager.go:712] Killing unwanted container "liveness"(id={"docker" "95567f85708ffac8b34b6c6f2bdb49d8eb57e7704b7b416083c7f296dd40cd0b"}) for pod "liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a)"
测试总时间:
$ kubectl get po -w
NAME READY STATUS RESTARTS AGE
liveness-http 0/1 Running 21 99m
liveness-http 0/1 CrashLoopBackOff 21 101m
liveness-http 0/1 Running 22 106m
liveness-http 1/1 Running 22 106m
liveness-http 0/1 Running 22 106m
liveness-http 0/1 Running 23 109m
liveness-http 1/1 Running 23 109m
liveness-http 0/1 Running 23 109m
liveness-http 0/1 CrashLoopBackOff 23 112m
liveness-http 0/1 Running 24 117m
liveness-http 1/1 Running 24 117m
liveness-http 0/1 Running 24 117m
结论
Liveness probe check didn't not called anymore
Liveness check
在 Kubernetes 创建 pod 时创建,并在每次
Pod
重新启动时重新创建。在您的配置中,您已设置
initialDelaySeconds: 20
,因此在创建
pod
后,Kubernetes 将等待 20 秒,然后它将调用
liveness
探测器 3 次(因为您已设置
failureThreshold: 3
)。 3次失败后,Kubernetes会根据
RestartPolicy
重启这个pod。同样在日志中,您将能够在日志中找到:
Mar 09 16:14:58 kubeadm kubelet[3855]: I0309 16:14:58.465470 3855 kuberuntime_manager.go:656] Container "liveness" ({"docker" "95567f85708ffac8b34b6c6f2bdb4
9d8eb57e7704b7b416083c7f296dd40cd0b"}) of pod liveness-http_default(8c87a08e-34aa-4bb1-be9b-fdca39a4562a): Container liveness failed liveness probe, will be restarted
为什么会重启?答案可以在
Container probes 中找到。
livenessProbe:
Indicates whether the container is running. If the liveness probe fails, the kubelet kills the container, and the container is subjected to its restart policy.
GKE
中的默认
Restart Policy 是
Always
。因此,您的 pod 将一遍又一遍地重新启动。
Readiness probe check called but interval became long ( maximum interval looks about 10 minutes)
$ kubectl get events
和
$ kubectl describe po
得出了这个结论。在这两种情况下,默认事件都会在 1 小时后删除。在我的
Tests
部分,您可以看到
Readiness probe
条目从
14:48:34
到
16:04:24
,因此 Kubernetes 每 10 秒调用一次
Readiness Probe
。
Why Liveness probe didn't run, and interval of Readiness probe changed?
Tests
部分向您展示的那样,
Readiness probe
没有改变。在这种情况下误导是使用
$ kubectl events
。关于
Liveiness Probe
它仍在调用,但在 pod 之后只有 3 次是
created
/
restarted
。我还包括了
$ kubectl get po -w
的输出。重新创建
pod
时,您可能会在 kubelet 日志中找到那些
liveness probes
。
In my plan, I would create alert policy, whose condition is like:
- if liveness probe error happens 3 times in 3 minutes
liveness probe
失败 3 次,则使用您当前的设置,它将重新启动此 pod。在这种情况下,您可以使用每个
restart
来创建一个
alert
。
Metric: kubernetes.io/container/restart_count
Resource type: k8s_container
您可以在 Stackoverflow 案例中找到一些关于
Monitoring alert
的有用信息,例如:
关于google-kubernetes-engine - Liveness Probe、Readiness Probe 未在预期持续时间内调用,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/66299478/
我对java有点陌生,所以如果我犯了一个简单的错误,请原谅我,但我不确定我哪里出错了,我收到的错误是“预期的.class,预期的标识符,而不是声明, ';'预期的。”我尝试了不同的方法,并从这些方法中
This question already has answers here: chai test array equality doesn't work as expected (3个答案) 3年前
我正在学习 Java(对不起,我的英语很差,这不是我的母语),当我在 Eclipse (JavaSE-1.7) 中在我输入的每个“try”中执行“try-finally” block 时,会出现以下消
我收到两个错误,指出 token 上的语法错误,ConstructorHeaderName expected instead & token “(”上的语法错误,< expected 在线: mTM.
我找不到错误。 Eclipse 给我这个错误。每个 { } 都是匹配的。请帮忙。 Multiple markers at this line - Syntax error on token “)”,
代码: import java.awt.*; import javax.swing.*; import java.awt.event.*; public class DoubleIt extends
我正在用 python(Vs 代码)编写代码,但出现此错误: Expected ")" Pylance 错误发生在:def main() 我试着运行我的 main 并将它打印到我的屏幕上。我用谷歌搜
我正在尝试按照 documentation 中的建议使用异步函数。但我收到此错误 意外的 token ,预期 ( async function getMoviesFromApi() { try
Closed. This question does not meet Stack Overflow guidelines。它当前不接受答案。 想改善这个问题吗?更新问题,以便将其作为on-topic
Closed. This question does not meet Stack Overflow guidelines。它当前不接受答案。 想改善这个问题吗?更新问题,以便将其作为on-topic
第一行包含一个表示数组长度的整数p。第二行包含用空格分隔的整数,这些整数描述数组中的每个元素。第三行打印一个整数,指示负数组的数量。 package asgn3; import java.util.*
好的,我是初学者,我必须修复此 java 表达式语言代码才能在我的系统 (Windchill) 中工作,但看起来我在语法中遗漏了一些内容: LWCNormalizedObject lwc =
我无法编译我的程序! 我想我缺少一个花括号,但我怎么也看不出在哪里! import javax.swing.*; import java.awt.*;
我的 jQuery 代码有问题,我的 Firebug 向我发出警告:需要选择器。 这是代码: $("img[id$='_tick']").each(function() { $(this).c
我的新类(class) Fountainofyouth 遇到了问题。尝试构建整个项目后,调试器显示 warning: extended initializer lists only available
我已经从 Java 转向 CPP,并且正在努力围绕构造构造函数链进行思考,我认为这是我的问题的根源。 我的头文件如下: public: GuidedTour(); GuidedTour(string
鉴于以下 for(var i=0; i< data.cats.length; i++) list += buildCategories(data.cats[i]); jsLint 告诉我 Expect
我有这个 json,但 Visual Studio Code 在标题中给了我警告。 [ { "title": "Book A", "imageUrl": "https:
我正在尝试编写一个有条件地禁用四个特殊成员函数(复制构造、移动构造、复制赋值和移动赋值)的包装类,下面是我用于测试目的的快速草稿: enum class special_member : uint8_
所以我用 F# 编写了一个非常简单的程序,它应该对 1000 以下的所有 3 和 5 的倍数求和: [1..999] |> List.filter (fun x -> x % 3 = 0 || x %
我是一名优秀的程序员,十分优秀!