service - 看不到添加到 Prometheus Operator 服务监视器的目标

Question

我正在尝试设置目标以将目标添加到我的 Prometheus Operator 服务监控器(在我的 terraform 中，它使用 helm chart 来部署 prometheus、prometheus operator 和服务监控器以及一堆东西)。我成功部署服务监视器后，在prometheus中看不到新目标app.kubernetes.io/instance: jobs-manager。我不确定我在配置中做错了什么。我也在检查这个 document看看缺少什么但还不能弄清楚。下面是一些相关的配置文件:

1. /helm/charts/prometheus-abcd/templates/service_monitor.tpl

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: jobs-manager-servicemonitor
  # Change this to the namespace the Prometheus instance is running in
  namespace: prometheus
  labels:
    app: jobs-manager
    release: prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/instance: jobs-manager # Targets jobs-manager service
  endpoints:
  - port: http
    interval: 15s

1. /helm/charts/prometheus-abcd/Chart.yaml

#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

apiVersion: v1
appVersion: "1.0.0"
description: Prometheus Service monitor, customized for abcd
name: prometheus-abcd
version: 1.0.0

1. /terraform/kubernetes/helm_values/prometheus.yaml

prometheus:
  podMetadata:
    annotations:
      container.apparmor.security.beta.kubernetes.io/prometheus-operator: runtime/default
      seccomp.security.alpha.kubernetes.io/pod: runtime/default

nodeAffinityPreset:
  ## Node affinity type
  ## Allowed values: soft, hard
  ##
  type: "hard"
  ## Node label key to match
  ## E.g.
  ## key: "kubernetes.io/e2e-az-name"
  ##
  key: "cloud.google.com/gke-nodepool"
  ## Node label values to match
  ## E.g.
  ## values:
  ##   - e2e-az1
  ##   - e2e-az2
  ##
  values: [
    "abcd-primary-pool"
  ]

prometheus:
  configMaps:
    - prometheus-config

## ServiceMonitors to be selected for target discovery.
## If {}, select all ServiceMonitors
##
serviceMonitorSelector: {
  jobs-manager-servicemonitor
}
# matchLabels:
#   foo: bar

## Namespaces to be selected for ServiceMonitor discovery.
## See https://github.com/prometheusoperator/prometheusoperator/blob/master/
## Documentation/api.md#namespaceselector for usage
##
serviceMonitorNamespaceSelector: {
  matchNames: prometheus
}

运行此命令时:kubectl get -n prometheus prometheuses.monitoring.coreos.com prometheus-kube-prometheus-prometheus可以看到服务监视器部署成功:

但是当我运行这个命令时:kubectl describe -n prometheus prometheuses.monitoring.coreos.com prometheus-kube-prometheus-prometheus我看到许多参数仍然有缺失值，例如 serviceMonitorSelector

Name:         prometheus-kube-prometheus-prometheus
Namespace:    prometheus
Labels:       app.kubernetes.io/component=prometheus
              app.kubernetes.io/instance=prometheus
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=kube-prometheus
              helm.sh/chart=kube-prometheus-3.4.0
Annotations:  meta.helm.sh/release-name: prometheus
              meta.helm.sh/release-namespace: prometheus
API Version:  monitoring.coreos.com/v1
Kind:         Prometheus
Metadata:
  Creation Timestamp:  2021-05-26T15:19:42Z
  Generation:          1
  Managed Fields:
    API Version:  monitoring.coreos.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:meta.helm.sh/release-name:
          f:meta.helm.sh/release-namespace:
        f:labels:
          .:
          f:app.kubernetes.io/component:
          f:app.kubernetes.io/instance:
          f:app.kubernetes.io/managed-by:
          f:app.kubernetes.io/name:
          f:helm.sh/chart:
      f:spec:
        .:
        f:affinity:
          .:
          f:podAntiAffinity:
            .:
            f:preferredDuringSchedulingIgnoredDuringExecution:
        f:alerting:
          .:
          f:alertmanagers:
        f:configMaps:
        f:enableAdminAPI:
        f:externalUrl:
        f:image:
        f:listenLocal:
        f:logFormat:
        f:logLevel:
        f:paused:
        f:podMetadata:
          .:
          f:labels:
            .:
            f:app.kubernetes.io/component:
            f:app.kubernetes.io/instance:
            f:app.kubernetes.io/name:
        f:podMonitorNamespaceSelector:
        f:podMonitorSelector:
        f:probeNamespaceSelector:
        f:probeSelector:
        f:replicas:
        f:retention:
        f:routePrefix:
        f:ruleNamespaceSelector:
        f:ruleSelector:
        f:securityContext:
          .:
          f:fsGroup:
          f:runAsUser:
        f:serviceAccountName:
        f:serviceMonitorNamespaceSelector:
        f:serviceMonitorSelector:
    Manager:         Go-http-client
    Operation:       Update
    Time:            2021-05-26T15:19:42Z
  Resource Version:  11485229
  Self Link:         /apis/monitoring.coreos.com/v1/namespaces/prometheus/prometheuses/prometheus-kube-prometheus-prometheus
  UID:               xxxxxxxxxxxxxxxxxxxx
Spec:
  Affinity:
    Pod Anti Affinity:
      Preferred During Scheduling Ignored During Execution:
        Pod Affinity Term:
          Label Selector:
            Match Labels:
              app.kubernetes.io/component:  prometheus
              app.kubernetes.io/instance:   prometheus
              app.kubernetes.io/name:       kube-prometheus
          Namespaces:
            prometheus
          Topology Key:  kubernetes.io/hostname
        Weight:          1
  Alerting:
    Alertmanagers:
      Name:         prometheus-kube-prometheus-alertmanager
      Namespace:    prometheus
      Path Prefix:  /
      Port:         http
  Config Maps:
    prometheus-config
  Enable Admin API:  false
  External URL:      http://prometheus-kube-prometheus-prometheus.prometheus:9090/
  Image:             docker.io/bitnami/prometheus:2.24.0-debian-10-r1
  Listen Local:      false
  Log Format:        logfmt
  Log Level:         info
  Paused:            false
  Pod Metadata:
    Labels:
      app.kubernetes.io/component:  prometheus
      app.kubernetes.io/instance:   prometheus
      app.kubernetes.io/name:       kube-prometheus
  Pod Monitor Namespace Selector:
  Pod Monitor Selector:
  Probe Namespace Selector:
  Probe Selector:
  Replicas:      1
  Retention:     10d
  Route Prefix:  /
  Rule Namespace Selector:
  Rule Selector:
  Security Context:
    Fs Group:            1001
    Run As User:         1001
  Service Account Name:  prometheus-kube-prometheus-prometheus
  Service Monitor Namespace Selector:
  Service Monitor Selector:
Events:  <none>

这就是我检查这个 document 的原因获取 serviceMonitorSelector 和 serviceMonitorNamespaceSelector 的模板，并将它们添加到上面的 prometheus.yaml 文件中，但不确定是否正确添加。

任何有使用 helm 和 terraform 设置服务监视器的经验的人，你能帮我检查一下我做错了什么吗？提前谢谢你。

Answer 1

你在 prometheus.yaml 中传递值的方式是错误的

serviceMonitorNamespaceSelector: {
  matchNames: prometheus
}                                  #this is wrong way

您应该设置如下值:

serviceMonitorNamespaceSelector:
       matchLabels:
         prometheus: somelabel

也一样

serviceMonitorSelector: {
  jobs-manager-servicemonitor
}

没有设置正确的方式。

引用请查看:https://github.com/prometheus-community/helm-charts/blob/83aa113f52e5f45fd04b4dd909172a6da1826592/charts/kube-prometheus-stack/values.yaml#L2034

检查这个很好的例子:https://rtfm.co.ua/en/kubernetes-a-clusters-monitoring-with-the-prometheus-operator/

使用 Terraform 和 helm 的普罗米修斯运算符(operator):https://github.com/OpenQAI/terraform-helm-release-prometheus-operator