Gitlab 中的 Redis 无法启动(断言失败的对象上下文)

Question

我有一个 Gitlab 安装，今天它停止工作了。挖掘之后，我发现 Redis 没有加载(所以 gitlab 失败)。

如果我运行 redis 服务器，它工作正常:

$ /opt/gitlab/embedded/bin/redis-server
23315:C 24 May 13:22:54.874 # Warning: no config file specified, using the default config. In order to specify a config file use /opt/gitlab/embedded/bin/redis-server /path/to/redis.conf
....
23315:M 24 May 13:22:54.875 * DB loaded from disk: 0.001 seconds
23315:M 24 May 13:22:54.876 * The server is now ready to accept connections on port 6379

但是，一旦我使用 giltab 配置文件启动它，它就会因段错误而失败。

$ /opt/gitlab/embedded/bin/redis-server /var/opt/gitlab/redis/redis.conf
23569:M 24 May 13:23:38.930 * Increased maximum number of open files to 10032 (it was originally set to 1024).
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 3.2.11 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 0
 |    `-._   `._    /     _.-'    |     PID: 23569
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

23569:M 24 May 13:23:38.931 # Server started, Redis version 3.2.11
23569:M 24 May 13:23:38.931 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
23569:M 24 May 13:23:38.931 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.


=== REDIS BUG REPORT START: Cut & paste starting from here ===
23569:M 24 May 13:23:38.932 # === ASSERTION FAILED OBJECT CONTEXT ===
23569:M 24 May 13:23:38.932 # Object type: 0
23569:M 24 May 13:23:38.932 # Object encoding: 0
23569:M 24 May 13:23:38.932 # Object refcount: 1
23569:M 24 May 13:23:38.932 # Object raw string len: 0
23569:M 24 May 13:23:38.932 # Object raw string content: ""
23569:M 24 May 13:23:38.932 # === ASSERTION FAILED ===
23569:M 24 May 13:23:38.933 # ==> db.c:161 'retval == DICT_OK' is not true
23569:M 24 May 13:23:38.933 # (forcing SIGSEGV to print the bug report.)
23569:M 24 May 13:23:38.933 # Redis 3.2.11 crashed by signal: 11
23569:M 24 May 13:23:38.933 # Crashed running the instuction at: 0x45d12a
23569:M 24 May 13:23:38.933 # Accessing address: 0xffffffffffffffff
23569:M 24 May 13:23:38.933 # Failed assertion: retval == DICT_OK (db.c:161)

------ STACK TRACE ------
EIP:
/opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(_serverAssert+0x6a)[0x45d12a]

Backtrace:
/opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(logStackTrace+0x29)[0x45edd9]
/opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(sigsegvHandler+0xac)[0x45f4dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f17e1cf5390]
/opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(_serverAssert+0x6a)[0x45d12a]
/opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(dbAdd+0x7f)[0x43aa6f]
/opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(rdbLoad+0x246)[0x443fe6]
/opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(loadDataFromDisk+0x34)[0x42b3c4]
/opt/gitlab/embedded/bin/redis-server 127.0.0.1:0(main+0x381)[0x41e941]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f17e193a830]
/opt/gitlab/embedded/bin/redis-server 127.0.0.1:0[0x41ec42]

------ INFO OUTPUT ------
# Server
redis_version:3.2.11
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:e819ac60ab48b2b9
redis_mode:standalone
os:Linux 4.4.0-127-generic x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.4
process_id:23569
run_id:7f57a86c15e8ee07ad0e5ee8198e2ef5d41ce631
tcp_port:0
uptime_in_seconds:0
uptime_in_days:0
hz:10
lru_clock:434362
executable:/opt/gitlab/embedded/bin/redis-server
config_file:/var/opt/gitlab/redis/redis.conf

# Clients
connected_clients:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:1026496
used_memory_human:1002.44K
used_memory_rss:0
used_memory_rss_human:0B
used_memory_peak:1026496
used_memory_peak_human:1002.44K
total_system_memory:2097258496
total_system_memory_human:1.95G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:0.00
mem_allocator:jemalloc-4.0.3

# Persistence
loading:1
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1527161018
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
loading_start_time:1527161018
loading_total_bytes:457327
loading_loaded_bytes:0
loading_loaded_perc:0.00
loading_eta_seconds:1

# Stats
total_connections_received:0
total_commands_processed:0
instantaneous_ops_per_sec:0
total_net_input_bytes:0
total_net_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0

# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.00
used_cpu_user:0.00
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

# Commandstats

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=1090,expires=486,avg_ttl=0
hash_init_value: 1527304277

------ CLIENT LIST OUTPUT ------

------ REGISTERS ------
23569:M 24 May 13:23:38.934 #
RAX:0000000000000000 RBX:00000000000000a1
RCX:000000007fffffb7 RDX:0000000000000000
RDI:00007f17e1cdf620 RSI:00007f17e1ce0780
RBP:00000000004d6f7d RSP:00007ffe606e21e0
R8 :00007f17e1ce0780 R9 :00007f17e2625780
R10:000000000000002a R11:0000000000000000
R12:00000000004d4cfe R13:0000000000000000
R14:0000000000000000 R15:00007f17e102c570
RIP:000000000045d12a EFL:0000000000010202
CSGSFS:0000000000000033
23569:M 24 May 13:23:38.934 # (00007ffe606e21ef) -> 0000000000470ad0
23569:M 24 May 13:23:38.934 # (00007ffe606e21ee) -> 0000000000470ab0
23569:M 24 May 13:23:38.934 # (00007ffe606e21ed) -> 00000163c5647d00
23569:M 24 May 13:23:38.934 # (00007ffe606e21ec) -> 0000000000000001
23569:M 24 May 13:23:38.934 # (00007ffe606e21eb) -> 0000000000000007
23569:M 24 May 13:23:38.934 # (00007ffe606e21ea) -> 0000016391e3da33
23569:M 24 May 13:23:38.934 # (00007ffe606e21e9) -> 00007f17e102c570
23569:M 24 May 13:23:38.934 # (00007ffe606e21e8) -> 0000000000000002
23569:M 24 May 13:23:38.934 # (00007ffe606e21e7) -> 0000000000443fe6
23569:M 24 May 13:23:38.934 # (00007ffe606e21e6) -> 00000000023e5430
23569:M 24 May 13:23:38.934 # (00007ffe606e21e5) -> 00007f17e1027800
23569:M 24 May 13:23:38.934 # (00007ffe606e21e4) -> ffffffffffffffff
23569:M 24 May 13:23:38.934 # (00007ffe606e21e3) -> 000000000043aa6f
23569:M 24 May 13:23:38.934 # (00007ffe606e21e2) -> 00007f17e1027800
23569:M 24 May 13:23:38.934 # (00007ffe606e21e1) -> 00007f17e102c570
23569:M 24 May 13:23:38.934 # (00007ffe606e21e0) -> 00007f17e11785e8

------ FAST MEMORY TEST ------
23569:M 24 May 13:23:38.934 # Bio thread for job type #0 terminated
23569:M 24 May 13:23:38.934 # Bio thread for job type #1 terminated
*** Preparing to test memory region 718000 (94208 bytes)
*** Preparing to test memory region 23d7000 (135168 bytes)
*** Preparing to test memory region 7f17dffff000 (8388608 bytes)
*** Preparing to test memory region 7f17e0800000 (10485760 bytes)
*** Preparing to test memory region 7f17e1600000 (2097152 bytes)
*** Preparing to test memory region 7f17e1ce0000 (16384 bytes)
*** Preparing to test memory region 7f17e1efd000 (16384 bytes)
*** Preparing to test memory region 7f17e2625000 (20480 bytes)
*** Preparing to test memory region 7f17e2635000 (4096 bytes)
.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: _serverAssert (base: 0x45d0c0)
Module: /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0 (base 0x400000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x45d0c0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
23569:M 24 May 13:23:39.001 # dump of function (hexdump of 234 bytes):
41548b05f4c72b004989fc554889f585c05389d37505e8a5fdffffbeb6b54d00bf0300000031c0e8247cfcff4d89e089d94889eabecfb54d00bf0300000031c0e80b7cfcffbe60a74d00bf0300000031c04c892590c72b0048892d91c72b00891d93c72b00e8e67bfcffc60425ffffffff785b5d415cc3660f1f84000000000041544989fc55534881eca00000008b1568c72b0064488b042528000000488984249800000031c085d20f842101000031c0be90a74d00bf03000000e8907bfcff418b94249800000031c0beeab54d00bf0300000031ed31dbe8737bfcff418b54240831c0befdb54d00bf
Function at 0x424d10 is serverLog

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           http://github.com/antirez/redis/issues

  Suspect RAM error? Use redis-server --test-memory to verify it.

Segmentation fault (core dumped)

这是 redis.conf 文件(没有注释)。

bind 127.0.0.1
port 0
unixsocket /var/opt/gitlab/redis/redis.socket
unixsocketperm 777
timeout 60
tcp-keepalive 300
daemonize no
pidfile "/var/run/redis_0.pid"
loglevel notice
logfile ""
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
dir "/var/opt/gitlab/redis"
slave-serve-stale-data yes
slave-read-only yes
repl-disable-tcp-nodelay no
slave-priority 100
maxclients 10000
maxmemory 0
maxmemory-policy noeviction
maxmemory-samples 5
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

谢谢。

更新

好的，问题出在redis.conf 中的dir "/var/opt/gitlab/redis" 选项

但我无法编辑它，因为当我执行 gitlab-ctl reconfigure 时，它会再次更改其值。

更新 2

这似乎是一个权限问题。

Redis 以/opt/gitlab/sv/redis/run 中定义的 gitlab-redis 用户身份执行

#!/bin/sh
exec 2>&1

umask 077
exec chpst -P -U gitlab-redis -u gitlab-redis /opt/gitlab/embedded/bin/redis-server /var/opt/gitlab/redis/redis.conf

根据redis.conf，它会尝试将数据库保存在/opt/gitlab/sv/redis/dump.rdb。但是这个文件夹是root所有的，gitlab-redis在这个目录下没有任何权限。

更改目录权限并不能解决问题，只要我执行 gitlab-ctl reconfigure 它就会再次更改目录的权限。

Answer 1

已修复。

我真的不知道哪一步解决了这个问题，但我认为在删除(实际上是重命名)文件 /var/opt/gitlab/redis/dump.rdb