azure - 当 Pod 重新创建时，Prometheus 发生崩溃循环

Question

我们在 Azure 中使用的是 Prometheus 版本 2.26.0 和 kubernetes 版本 1.21.7。我们将数据装载到Azure 存储 NFS 中，并且运行良好。从最近几天开始，Prometheus 容器出现崩溃循环，下面是日志

level=info ts=2022-01-26T08:04:14.375Z caller=main.go:418 msg="Starting Prometheus" version="(version=2.26.0, branch=HEAD, revision=3cafc58827d1ebd1a67749f88be4218f0bab3d8d)"
level=info ts=2022-01-26T08:04:14.375Z caller=main.go:423 build_context="(go=go1.16.2, user=root@a67cafebe6d0, date=20210331-11:56:23)"
level=info ts=2022-01-26T08:04:14.375Z caller=main.go:424 host_details="(Linux 5.4.0-1065-azure #68~18.04.1-Ubuntu SMP Fri Dec 3 14:08:44 UTC 2021 x86_64 prometheus-6b9d9d54f4-nc45x (none))"
level=info ts=2022-01-26T08:04:14.375Z caller=main.go:425 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2022-01-26T08:04:14.375Z caller=main.go:426 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2022-01-26T08:04:14.503Z caller=web.go:540 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2022-01-26T08:04:14.507Z caller=main.go:795 msg="Starting TSDB ..."
level=info ts=2022-01-26T08:04:14.509Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
level=info ts=2022-01-26T08:04:14.560Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641478251052 maxt=1641513600000 ulid=01FRSEHC4YHV3N26JY5AMNZFRW
level=info ts=2022-01-26T08:04:14.593Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641513600037 maxt=1641578400000 ulid=01FRVCAP2VJGDF0Z9CS24EXAJJ
level=info ts=2022-01-26T08:04:14.624Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641578400038 maxt=1641643200000 ulid=01FRXA4AQHMHAEYWRKQFGP075M
level=info ts=2022-01-26T08:04:14.651Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641643200422 maxt=1641708000000 ulid=01FRZ7XQQ4RA96DCPPBP22D71N
level=info ts=2022-01-26T08:04:14.679Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641708000020 maxt=1641772800000 ulid=01FS15QDG6BS7H6M6Y09HG3E12
level=info ts=2022-01-26T08:04:14.707Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641772800011 maxt=1641837600000 ulid=01FS33GT38PRSB9VP56YFXT2M0
level=info ts=2022-01-26T08:04:14.736Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641963555381 maxt=1641967200000 ulid=01FS6MRNZEWT1Z6P697K09KHD7
level=info ts=2022-01-26T08:04:14.763Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641837600100 maxt=1641902400000 ulid=01FS6R88C70TCD8CYC4XJ95X23
level=info ts=2022-01-26T08:04:14.810Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641967200019 maxt=1642032000000 ulid=01FS8WXQP3YJ7EXBVNYBQG4DVY
level=info ts=2022-01-26T08:04:14.836Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642032000072 maxt=1642096800000 ulid=01FSATQBR4XBQRDM72ATFS9PQ2
level=info ts=2022-01-26T08:04:14.863Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642096800059 maxt=1642161600000 ulid=01FSCRHE2YBDX7GPRPSH6BNGRX
level=info ts=2022-01-26T08:04:14.895Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642161600091 maxt=1642226400000 ulid=01FSEPB1GPGAANVCQ2VKW9BQ4G
level=info ts=2022-01-26T08:04:14.948Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642226400026 maxt=1642291200000 ulid=01FSGM4J0G1D0A6H1GD3N9C372
level=info ts=2022-01-26T08:04:14.973Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642291200005 maxt=1642356000000 ulid=01FSJHY6W0FRYDHCXBVB5XPFYG
level=info ts=2022-01-26T08:04:15.002Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642356000027 maxt=1642420800000 ulid=01FSMFR96DASV6YPN66W7C86H9
level=info ts=2022-01-26T08:04:15.077Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642420800042 maxt=1642485600000 ulid=01FSPDHGWRT65D8CKWQ2JPRHW3
level=info ts=2022-01-26T08:04:15.105Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642485600006 maxt=1642550400000 ulid=01FSRBAVP2MW71H08F32D6HGB4
level=info ts=2022-01-26T08:04:15.130Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642550400028 maxt=1642615200000 ulid=01FST9482FD0Z3PHXHNW2W616E
level=info ts=2022-01-26T08:04:15.157Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642680000018 maxt=1642687200000 ulid=01FSW00TJKJ7CGCQ7JJS3XQK8G
level=info ts=2022-01-26T08:04:15.187Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642687200018 maxt=1642694400000 ulid=01FSW6WHTSEAXHWV5J7PQP94X7
level=info ts=2022-01-26T08:04:15.213Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642615200021 maxt=1642680000000 ulid=01FSW6XYH2Y429PG5YRM0K45XS
level=info ts=2022-01-26T08:04:15.275Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642694400018 maxt=1642701600000 ulid=01FSWDR92Y7H302NDZRX1V2PX9
level=info ts=2022-01-26T08:04:21.840Z caller=head.go:696 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2022-01-26T08:04:22.623Z caller=head.go:710 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=782.403397ms
level=info ts=2022-01-26T08:04:22.623Z caller=head.go:716 component=tsdb msg="Replaying WAL, this may take a while"
level=info ts=2022-01-26T08:04:34.169Z caller=head.go:742 component=tsdb msg="WAL checkpoint loaded"
level=info ts=2022-01-26T08:04:38.895Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=299 maxSegment=7511
level=warn ts=2022-01-26T08:04:46.423Z caller=main.go:645 msg="Received SIGTERM, exiting gracefully..."
level=info ts=2022-01-26T08:04:46.424Z caller=main.go:668 msg="Stopping scrape discovery manager..."
level=info ts=2022-01-26T08:04:46.424Z caller=main.go:682 msg="Stopping notify discovery manager..."
level=info ts=2022-01-26T08:04:46.424Z caller=main.go:704 msg="Stopping scrape manager..."
level=info ts=2022-01-26T08:04:46.424Z caller=main.go:678 msg="Notify discovery manager stopped"
level=info ts=2022-01-26T08:04:46.425Z caller=main.go:698 msg="Scrape manager stopped"
level=info ts=2022-01-26T08:04:46.426Z caller=manager.go:934 component="rule manager" msg="Stopping rule manager..."
level=info ts=2022-01-26T08:04:46.426Z caller=manager.go:944 component="rule manager" msg="Rule manager stopped"
level=info ts=2022-01-26T08:04:46.426Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."
level=info ts=2022-01-26T08:04:46.426Z caller=main.go:872 msg="Notifier manager stopped"
level=info ts=2022-01-26T08:04:46.426Z caller=main.go:664 msg="Scrape discovery manager stopped"
level=info ts=2022-01-26T08:04:46.792Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=300 maxSegment=7511
level=info ts=2022-01-26T08:04:46.870Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=301 maxSegment=7511
level=info ts=2022-01-26T08:04:46.901Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=302 maxSegment=7511
level=info ts=2022-01-26T08:04:46.946Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=303 maxSegment=7511
level=info ts=2022-01-26T08:04:46.974Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=304 maxSegment=7511
level=info ts=2022-01-26T08:04:47.008Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=305 maxSegment=7511
level=info ts=2022-01-26T08:04:47.034Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=306 maxSegment=7511
level=info ts=2022-01-26T08:04:47.067Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=307 maxSegment=7511
level=info ts=2022-01-26T08:04:47.098Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=308 maxSegment=7511
level=info ts=2022-01-26T08:04:47.124Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=309 maxSegment=7511
level=info ts=2022-01-26T08:04:47.158Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=310 maxSegment=7511
level=info ts=2022-01-26T08:04:47.203Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=311 maxSegment=7511
level=info ts=2022-01-26T08:04:47.254Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=312 maxSegment=7511
level=info ts=2022-01-26T08:04:47.486Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=313 maxSegment=7511
level=info ts=2022-01-26T08:04:47.511Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=314 maxSegment=7511
level=info ts=2022-01-26T08:04:47.539Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=315 maxSegment=7511
level=info ts=2022-01-26T08:04:47.564Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=316 maxSegment=7511

.
.
.
.
.
.
.
.
.

level=info ts=2022-01-26T08:05:15.161Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1401 maxSegment=7511
level=info ts=2022-01-26T08:05:15.182Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1402 maxSegment=7511
level=info ts=2022-01-26T08:05:15.205Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1403 maxSegment=7511
level=info ts=2022-01-26T08:05:15.229Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1404 maxSegment=7511
level=info ts=2022-01-26T08:05:15.251Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1405 maxSegment=7511
level=info ts=2022-01-26T08:05:15.274Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1406 maxSegment=7511
level=info ts=2022-01-26T08:05:15.297Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1407 maxSegment=7511
level=info ts=2022-01-26T08:05:15.323Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1408 maxSegment=7511
level=info ts=2022-01-26T08:05:15.349Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1409 maxSegment=7511
level=info ts=2022-01-26T08:05:15.372Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1410 maxSegment=7511
level=info ts=2022-01-26T08:05:15.426Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1411 maxSegment=7511
level=info ts=2022-01-26T08:05:15.452Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1412 maxSegment=7511
level=info ts=2022-01-26T08:05:15.475Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1413 maxSegment=7511
level=info ts=2022-01-26T08:05:15.498Z caller=head.go:768 component=tsdb msg="WAL segment loaded" segment=1414 maxSegment=7511
rpc error: code = NotFound desc = an error occurred when try to find container "ae14079418f59b04bb80d8413e8fdc34f167bfe762317ef674e05466d34c9e1f": not found

因此，我删除了部署并重新部署到同一存储帐户，但出现了新错误

level=info ts=2022-01-26T11:10:11.530Z caller=main.go:418 msg="Starting Prometheus" version="(version=2.26.0, branch=HEAD, revision=3cafc58827d1ebd1a67749f88be4218f0bab3d8d)"
level=info ts=2022-01-26T11:10:11.534Z caller=main.go:423 build_context="(go=go1.16.2, user=root@a67cafebe6d0, date=20210331-11:56:23)"
level=info ts=2022-01-26T11:10:11.535Z caller=main.go:424 host_details="(Linux 5.4.0-1064-azure #67~18.04.1-Ubuntu SMP Wed Nov 10 11:38:21 UTC 2021 x86_64 prometheus-6b9d9d54f4-wnmzh (none))"
level=info ts=2022-01-26T11:10:11.536Z caller=main.go:425 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2022-01-26T11:10:11.536Z caller=main.go:426 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2022-01-26T11:10:14.168Z caller=web.go:540 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2022-01-26T11:10:15.385Z caller=main.go:795 msg="Starting TSDB ..."
level=info ts=2022-01-26T11:10:16.022Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641837600024 maxt=1641902400000 ulid=01FS51ANKBFTVNRPZ68FGQQ5GA
level=info ts=2022-01-26T11:10:16.309Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
level=info ts=2022-01-26T11:10:16.494Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641902400005 maxt=1641967200000 ulid=01FS6Z46FGXN932K7D39D9166D
level=info ts=2022-01-26T11:10:16.806Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1641967200106 maxt=1642032000000 ulid=01FS8WXRJ7Q80FKD4C8EJNR0AD
level=info ts=2022-01-26T11:10:17.011Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642032000003 maxt=1642096800000 ulid=01FSATQE1VMNR101KRW1X10Q75
level=info ts=2022-01-26T11:10:17.305Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642096800206 maxt=1642161600000 ulid=01FSCRGVT1E7562SF7EQN12JBM
level=info ts=2022-01-26T11:10:18.240Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642161600059 maxt=1642226400000 ulid=01FSEPAFP2CX03ANRB7Q1AG514
level=info ts=2022-01-26T11:10:21.046Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642226400051 maxt=1642291200000 ulid=01FSGM3WT0TKR0XW9BD4QSKPQE
level=info ts=2022-01-26T11:10:21.422Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642291200113 maxt=1642356000000 ulid=01FSJHXKHMANW0E6FXDXVM265G
level=info ts=2022-01-26T11:10:22.822Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642356000032 maxt=1642420800000 ulid=01FSMFQ6XJ97VJFKNCYQBVB4DZ
level=info ts=2022-01-26T11:10:23.536Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642420800021 maxt=1642485600000 ulid=01FSPDGM95FDDV2CDWX93BTDCS
level=info ts=2022-01-26T11:10:23.880Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642485600072 maxt=1642550400000 ulid=01FSRBA555RWY4QNP4HD9YKRBM
level=info ts=2022-01-26T11:10:25.021Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642550400031 maxt=1642615200000 ulid=01FST93N3C82K9VS20MKTMGGYC
level=info ts=2022-01-26T11:10:25.713Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642615200014 maxt=1642680000000 ulid=01FSW6X95FRNSN1XJZ2YK0MXW7
level=info ts=2022-01-26T11:10:26.634Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642680000012 maxt=1642744800000 ulid=01FSY4PXA7V1XQHHA3MC35JSWQ
level=info ts=2022-01-26T11:10:27.776Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642744800174 maxt=1642809600000 ulid=01FT02G9XGHPV8GME53ZPMYXE6
level=info ts=2022-01-26T11:10:28.760Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642809600070 maxt=1642874400000 ulid=01FT209WP8AXXVZB1NCSC55ACE
level=info ts=2022-01-26T11:10:29.618Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642874400253 maxt=1642939200000 ulid=01FT3Y3A4H72FFW318RKHEXXGA
level=info ts=2022-01-26T11:10:30.313Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1642939200047 maxt=1643004000000 ulid=01FT5VX3YC838QN5VQFAERV1QX
level=info ts=2022-01-26T11:10:30.483Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1643004000040 maxt=1643068800000 ulid=01FT7SPHC5EV0SS1R0WT04H9FR
level=info ts=2022-01-26T11:10:30.696Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1643068800035 maxt=1643133600000 ulid=01FT9QFZXBZ7EYY2CTE8WXZTB9
level=info ts=2022-01-26T11:10:31.838Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1643133600000 maxt=1643155200000 ulid=01FTA574G4M45WX97Z470DQF73
level=info ts=2022-01-26T11:10:33.686Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1643176800008 maxt=1643184000000 ulid=01FTASSZCG8V5N2VGAGFBYJBSR
level=info ts=2022-01-26T11:10:36.078Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1643184000000 maxt=1643191200000 ulid=01FTB0NP47JW5JCF808QZZ8WZQ
level=info ts=2022-01-26T11:10:36.442Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1643155200065 maxt=1643176800000 ulid=01FTB0P9H3H09B2ADD5X1RXFW6
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:668 msg="Stopping scrape discovery manager..."
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:682 msg="Stopping notify discovery manager..."
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:704 msg="Stopping scrape manager..."
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:678 msg="Notify discovery manager stopped"
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:664 msg="Scrape discovery manager stopped"
level=info ts=2022-01-26T11:10:40.079Z caller=main.go:698 msg="Scrape manager stopped"
level=info ts=2022-01-26T11:10:40.080Z caller=manager.go:934 component="rule manager" msg="Stopping rule manager..."
level=info ts=2022-01-26T11:10:40.080Z caller=manager.go:944 component="rule manager" msg="Rule manager stopped"
level=info ts=2022-01-26T11:10:40.080Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."
level=info ts=2022-01-26T11:10:40.080Z caller=main.go:872 msg="Notifier manager stopped"
level=error ts=2022-01-26T11:10:40.080Z caller=main.go:881 err="opening storage failed: lock DB directory: resource temporarily unavailable"

yaml由Istio提供。下面是部署yaml文件。

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    component: "server"
    app: prometheus
    release: prometheus
    chart: prometheus-14.6.1
    heritage: Helm
  name: prometheus
  namespace: istio-system
spec:
  selector:
    matchLabels:
      component: "server"
      app: prometheus
      release: prometheus
  replicas: 1
  template:
    metadata:
      labels:
        component: "server"
        app: prometheus
        release: prometheus
        chart: prometheus-14.6.1
        heritage: Helm
        
        sidecar.istio.io/inject: "false"
    spec:
      enableServiceLinks: true
      serviceAccountName: prometheus
      containers:
        - name: prometheus-server-configmap-reload
          image: "jimmidyson/configmap-reload:v0.5.0"
          imagePullPolicy: "IfNotPresent"
          args:
            - --volume-dir=/etc/config
            - --webhook-url=http://127.0.0.1:9090/-/reload
          resources:
            {}
          volumeMounts:
            - name: config-volume
              mountPath: /etc/config
              readOnly: true

        - name: prometheus-server
          image: "prom/prometheus:v2.26.0"
          imagePullPolicy: "IfNotPresent"
          args:
            - --storage.tsdb.retention.time=15d
            - --config.file=/etc/config/prometheus.yml
            - --storage.tsdb.path=/data
            - --web.console.libraries=/etc/prometheus/console_libraries
            - --web.console.templates=/etc/prometheus/consoles
            - --web.enable-lifecycle
          ports:
            - containerPort: 9090
          readinessProbe:
            httpGet:
              path: /-/ready
              port: 9090
            initialDelaySeconds: 0
            periodSeconds: 5
            timeoutSeconds: 4
            failureThreshold: 3
            successThreshold: 1
          livenessProbe:
            httpGet:
              path: /-/healthy
              port: 9090
            initialDelaySeconds: 30
            periodSeconds: 15
            timeoutSeconds: 10
            failureThreshold: 3
            successThreshold: 1
          resources:
            {}
          volumeMounts:
            - name: config-volume
              mountPath: /etc/config
            - name: azurefileshare
              mountPath: /data
              subPath: ""
      hostNetwork: false
      dnsPolicy: ClusterFirst
      securityContext:
        fsGroup: 65534
        runAsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
      terminationGracePeriodSeconds: 300
      volumes:
        - name: config-volume
          configMap:
            name: prometheus
        - name: azurefileshare
          azureFile:
            secretName: log-storage-secret
            shareName: prometheusfileshare
            readOnly: false

预期行为当我将数据装载到新容器时，它应该加载数据。

实际行为旧 Pod 死亡后无法加载数据或无法将数据与新创建的 Pod 绑定(bind)

帮我解决这个问题。

Answer 1

谢谢 YwH对于您的建议，请发布此答案，以便在其他社区成员将来遇到相同问题时可以为他们提供帮助。

如本 document 中所述Istio 提供了基本的示例安装来快速启动和运行 Prometheus:这仅用于演示，并未针对性能或安全性进行调整。

Note : Isio configuration is well-suited for small clusters andmonitoring for short time horizons, it is not suitable for large-scalemeshes or monitoring over a period of days or weeks

解决方案:Prometheus 是一个有状态应用程序，最好使用 StatefulSet 进行部署，而不是 Deployment。

StatefulSet 对于需要以下一项或多项的应用程序非常有值(value)。

稳定、持久的存储。有序、优雅的部署和扩展。

您可以使用此 Stateful prometheus容器部署代码。