Prometheus 服务启动失败

Question

我过去 15 天一直在使用 Prometheus。从昨天开始，服务由于内存不足而失败。

我已将内存从 4GB 增加到 8GB，但现在我无法启动 Prometheus 服务并且失败并出现以下错误:

Nov 15 10:25:20 prometheus: level=info ts=2019-11-15T18:25:20.393Z caller=head.go:598 component=tsdb msg="head GC completed" duration=22.375774ms
Nov 15 10:25:35 prometheus: level=info ts=2019-11-15T18:25:35.135Z caller=head.go:668 component=tsdb msg="WAL checkpoint complete" first=549 last=550 duration=14.741502221s
Nov 15 10:25:35 prometheus: level=info ts=2019-11-15T18:25:35.795Z caller=head.go:598 component=tsdb msg="head GC completed" duration=22.593533ms
Nov 15 10:25:36 prometheus: level=info ts=2019-11-15T18:25:36.796Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50E9ZP9F1ZSY2YD30GEHK duration=732.572029ms
Nov 15 10:25:37 prometheus: level=error ts=2019-11-15T18:25:37.062Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50E9ZP9F1ZSY2YD30GEHK:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:38 prometheus: level=info ts=2019-11-15T18:25:38.824Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50G8EYTYBPWWHWZ1RJV5A duration=761.592662ms
Nov 15 10:25:39 prometheus: level=error ts=2019-11-15T18:25:39.087Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50G8EYTYBPWWHWZ1RJV5A:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:41 prometheus: level=info ts=2019-11-15T18:25:41.816Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50K6ZJFJQ4217JW2E5XBV duration=729.049353ms
Nov 15 10:25:42 prometheus: level=error ts=2019-11-15T18:25:42.087Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50K6ZJFJQ4217JW2E5XBV:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:46 prometheus: level=info ts=2019-11-15T18:25:46.814Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50R37245ZE56EZNH0335G duration=727.559705ms
Nov 15 10:25:47 prometheus: level=error ts=2019-11-15T18:25:47.084Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50R37245ZE56EZNH0335G:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:55 prometheus: level=info ts=2019-11-15T18:25:55.807Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR510WDE18F41GJBRVXBEDE duration=722.624112ms
Nov 15 10:25:55 prometheus: level=error ts=2019-11-15T18:25:56.080Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR510WDE18F41GJBRVXBEDE:mmap files: mmap: cannot allocate memory]"

普罗米修斯服务:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

我在同一台服务器上安装了 influxdb grafana。

请让我知道究竟是什么问题？

提前致谢

Answer 1

你运行的是 32 位的 prometheus 吗？您可能已经用完了地址空间。我不认为 Prometheus 计划在 32 位系统上进行这项工作，它只是用完了所有 mmap 的 4GB 地址空间。正在发生。

更多详情请见 https://github.com/prometheus/prometheus/issues/4392#issuecomment-433721793