kubernetes - 有没有办法使用 prometheus 监控 kube cron 作业

Question

有没有办法监控 kube cronjob？

我有一个 kube cronjob，它在我的集群上每 10 分钟运行一次。有没有办法在每次我的 cronjob 由于某些错误而失败时收集指标，或者在我的 cronjob 在一定时间后尚未完成时发出通知？

Answer 1

我将这些规则与 kube-state-metrics 一起使用:

groups:
- name: job.rules
  rules:
  - alert: CronJobRunning
    expr: time() -kube_cronjob_next_schedule_time > 3600
    for: 1h
    labels:
      severity: warning
    annotations:
      description: CronJob {{$labels.namespaces}}/{{$labels.cronjob}} is taking more than 1h to complete
      summary: CronJob didn't finish after 1h

  - alert: JobCompletion
    expr: kube_job_spec_completions - kube_job_status_succeeded  > 0
    for: 1h
    labels:
      severity: warning
    annotations:
      description: Job completion is taking more than 1h to complete
        cronjob {{$labels.namespaces}}/{{$labels.job}}
      summary: Job {{$labels.job}} didn't finish to complete after 1h

  - alert: JobFailed
    expr: kube_job_status_failed  > 0
    for: 1h
    labels:
      severity: warning
    annotations:
      description: Job {{$labels.namespaces}}/{{$labels.job}} failed to complete
      summary: Job failed