prometheus - 更新选项时出错 : "expanding series: block: XXXXXXXXXXXXXXX : decode postings: invalid size"

Question

我在从普罗米修斯获取指标时遇到以下错误，无法在互联网上找到任何相关内容，也无法在日志中找到任何相关内容..请帮助这里出了什么问题

错误更新选项:“扩展系列: block :01G3T0PCNX96SX43TWVEGKTVG4:解码帖子:无效大小”

我在运行普罗米修斯，版本 2.35.0

Answer 1

我不确定这是否适用于您的情况，但为了我将来的引用，我会快速写下解决我的问题的方法。

如前所述:

I have prometheus deployed as bitnami helm chart on my on-site k8s cluster. The error happens ever since i try to persist the data (because otherwise the charts disappear if the pod is restarted). We use CIFS as a storage system.

进一步，当我查看 pod 的日志时，我发现了这个

kubectl logs prometheus-prometheus-kube-prometheus-prometheus-0

ts=2022-09-09T11:00:35.345Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYH2WTM9X3TQD2T52RPEV6: invalid magic number 0"
ts=2022-09-09T11:00:54.184Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYHP0HMJZ1KVB608P3TV4E duration=2.838654027s
ts=2022-09-09T11:00:54.317Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYHP0HMJZ1KVB608P3TV4E: invalid magic number 0"
ts=2022-09-09T11:01:29.216Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYJR5DA60D17F3W2VAFZAT duration=2.899332494s
ts=2022-09-09T11:01:29.453Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYJR5DA60D17F3W2VAFZAT: invalid magic number 0"
ts=2022-09-09T11:02:32.354Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYMNTDYWWENPT2KSSNWN3H duration=2.900614585s
ts=2022-09-09T11:02:32.494Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYMNTDYWWENPT2KSSNWN3H: invalid magic number 0"
ts=2022-09-09T11:03:35.407Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYPKCES6PDQYB2MPCWFG44 duration=2.913101338s
ts=2022-09-09T11:03:35.554Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYPKCES6PDQYB2MPCWFG44: invalid magic number 0"
ts=2022-09-09T11:04:38.999Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYRGZ3TPP8Q03X7CKVSWHQ duration=3.444085613s
ts=2022-09-09T11:04:39.314Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYRGZ3TPP8Q03X7CKVSWHQ: read symbols: invalid checksum"
ts=2022-09-09T11:05:42.805Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYTF7KEBV6SPBYPR5SE8V3 duration=3.490813666s
ts=2022-09-09T11:05:42.992Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYTF7KEBV6SPBYPR5SE8V3: invalid magic number 0"
ts=2022-09-09T11:06:46.245Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYWDDHPSMCAZ4XK34DWH7D duration=3.252076412s
ts=2022-09-09T11:06:46.392Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYWDDHPSMCAZ4XK34DWH7D: invalid magic number 0"
ts=2022-09-09T11:07:49.278Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYYBASWE0EC9DTPQD72H98 duration=2.884889018s
ts=2022-09-09T11:07:49.435Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYYBASWE0EC9DTPQD72H98: invalid magic number 0"
ts=2022-09-09T11:08:52.155Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGZ08WWSMSG6PBZATX6HT1S duration=2.719327278s
ts=2022-09-09T11:08:52.269Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGZ08WWSMSG6PBZATX6HT1S: invalid magic number 0"
ts=2022-09-09T11:09:55.542Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGZ268E5K4S07EZS7GM9J16 duration=3.271842684s
ts=2022-09-09T11:09:55.893Z caller=head.go:844 level=info component=tsdb msg="Head GC completed" duration=223.829164ms
ts=2022-09-09T11:09:55.912Z caller=checkpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=0 to_segment=2 mint=1662717600000
ts=2022-09-09T11:10:03.499Z caller=head.go:1013 level=info component=tsdb msg="WAL checkpoint complete" first=0 last=2 duration=7.590667466s

特别考虑:压缩失败；损坏的 block XXX 无效的魔数(Magic Number) 0

对于这种错误情况，搜索结果要多得多:

• https://github.com/prometheus/prometheus/issues/7397
• https://github.com/prometheus/prometheus/issues/10679

仅举几例。尤其是后者表明，这很可能与 prometheus 不支持您的特定文件系统有关，这很可能与 CIFS 相关。根据以前的经验，我知道安装选项很重要，所以我添加了

mfsymlinks 和 cache=none 到我们的存储类。确实解决了这个问题。

顺便说一句。感谢@Machavity 让这个线程保持“干净”