Aof改写后的Redis主从切换

Question

这个Redis Cluster有240个节点(120个master和120个slave)，并且长期运行良好。但是现在弄个Master Slave switch 差不多要几个小时。

我从 Redis 服务器获取了一些日志。

5c541d3a765e087af7775ba308f51ffb2aa5415110.12.28.165:650213306:M 08 Mar 18:55:02.597 * Background append only file rewriting started by pid 1539613306:M 08 Mar 18:55:41.636 # Cluster state changed: fail13306:M 08 Mar 18:55:45.321 # Connection with slave client id #112948 lost.13306:M 08 Mar 18:55:46.243 # Configuration change detected. Reconfiguring myself as a replica of afb6e012db58bd26a7c96182b04f0a2ba6a4576813306:S 08 Mar 18:55:47.134 * AOF rewrite child asks to stop sending diffs.15396:C 08 Mar 18:55:47.134 * Parent agreed to stop sending diffs. Finalizing AOF...15396:C 08 Mar 18:55:47.134 * Concatenating 0.02 MB of AOF diff received from parent.15396:C 08 Mar 18:55:47.135 * SYNC append only file rewrite performed15396:C 08 Mar 18:55:47.186 * AOF rewrite: 4067 MB of memory used by copy-on-write13306:S 08 Mar 18:55:47.209 # Cluster state changed: ok

5ac747878f881349aa6a62b179176ddf603e034c10.12.30.107:650022825:M 08 Mar 18:55:30.534 * FAIL message received from da493af5bb3d15fc563961de09567a47787881be about 5c541d3a765e087af7775ba308f51ffb2aa5415122825:M 08 Mar 18:55:31.440 # Failover auth granted to afb6e012db58bd26a7c96182b04f0a2ba6a45768 for epoch 32322825:M 08 Mar 18:55:41.587 * Background append only file rewriting started by pid 2362822825:M 08 Mar 18:56:24.200 # Cluster state changed: fail22825:M 08 Mar 18:56:30.002 # Connection with slave client id #382416 lost.22825:M 08 Mar 18:56:30.830 * FAIL message received from 0decbe940c6f4d4330fae5a9c129f1ad4932405d about 5ac747878f881349aa6a62b179176ddf603e034c22825:M 08 Mar 18:56:30.840 # Failover auth denied to d46f95da06cfcd8ea5eaa15efabff5bd5e99df55: its master is up22825:M 08 Mar 18:56:30.843 # Configuration change detected. Reconfiguring myself as a replica of d46f95da06cfcd8ea5eaa15efabff5bd5e99df5522825:S 08 Mar 18:56:31.030 * Clear FAIL state for node 5ac747878f881349aa6a62b179176ddf603e034c: slave is reachable again.22825:S 08 Mar 18:56:31.030 * Clear FAIL state for node 5c541d3a765e087af7775ba308f51ffb2aa54151: slave is reachable again.22825:S 08 Mar 18:56:31.294 # Cluster state changed: ok22825:S 08 Mar 18:56:31.595 * Connecting to MASTER 10.12.30.104:640422825:S 08 Mar 18:56:31.671 * MASTER  SLAVE sync started22825:S 08 Mar 18:56:31.671 * Non blocking connect for SYNC fired the event.22825:S 08 Mar 18:56:31.672 * Master replied to PING, replication can continue...22825:S 08 Mar 18:56:31.673 * Partial resynchronization not possible (no cached master)22825:S 08 Mar 18:56:31.691 * AOF rewrite child asks to stop sending diffs.

附上Aof重写后Redis Master Slave Swtich发生

这是这个集群的配置。

daemonize notcp-backlog 511timeout 0tcp-keepalive 60loglevel noticedatabases 16dir "/var/cachecloud/data"stop-writes-on-bgsave-error norepl-timeout 60repl-ping-slave-period 10repl-disable-tcp-nodelay norepl-backlog-size 10000000repl-backlog-ttl 7200slave-serve-stale-data yesslave-read-only yesslave-priority 100lua-time-limit 5000slowlog-log-slower-than 10000slowlog-max-len 128hash-max-ziplist-entries 512hash-max-ziplist-value 64list-max-ziplist-entries 512list-max-ziplist-value 64set-max-intset-entries 512zset-max-ziplist-entries 128zset-max-ziplist-value 64activerehashing yesclient-output-buffer-limit normal 0 0 0client-output-buffer-limit slave 512mb 128mb 60client-output-buffer-limit pubsub 32mb 8mb 60hz 10port 6401maxmemory 13000mbmaxmemory-policy volatile-lruappendonly yesappendfsync noappendfilename "appendonly-6401.aof"dbfilename "dump-6401.rdb"aof-rewrite-incremental-fsync yesno-appendfsync-on-rewrite yesauto-aof-rewrite-min-size 62500kbauto-aof-rewrite-percentage 86rdbcompression yesrdbchecksum yesrepl-diskless-sync norepl-diskless-sync-delay 5maxclients 10000hll-sparse-max-bytes 3000min-slaves-to-write 0min-slaves-max-lag 10aof-load-truncated yesnotify-keyspace-events ""bind 10.12.26.226protected-mode nocluster-enabled yescluster-node-timeout 15000cluster-slave-validity-factor 10cluster-migration-barrier 1cluster-config-file "nodes-6401.conf"cluster-require-full-coverage norename-command FLUSHDB ""rename-command FLUSHALL ""rename-command KEYS ""

在我的选项中，aof rewrite 不会影响 Redis 主线程。但这似乎使该节点不响应其他节点的 Ping。

Answer 1

检查 Linux 内核参数上的 THP(Transparent Huge pages)。因为 AOF 差异大小为 0.02MB，写时复制大小为 2067MB。