O failover de cluster do Redis / Sentinel resulta em “mestre de failover-anular-não-eleito”

Question

O failover de cluster do Redis / Sentinel resulta em “mestre de failover-anular-não-eleito”

#1 resposta do (2 votos)

2

Eu tenho uma configuração de cluster Redis de 2 nós.

[master] 192.168.56.102: Redis Master (: 6379), Redis Slave (: 6380), Sentinela (: 26379), Sentinel # 2 (: 26380)

[resgate] 192.168.56.103: Redis Master (: 6379), Redis Slave (: 6380), Sentinel (: 26379)

Cada instância do escravo é um escravo da instância do mestre na mesma máquina. Cada instância sentinela monitora ambas as instâncias principais.

Estou usando o acima em conjunto com o twemproxy (que não tem nada a ver com essa questão) e um script client-reconfig para atualizar a configuração do twemproxy para que o aplicativo continue funcionando.

Estou parando os servidores para ver o que está acontecendo e se tudo está funcionando corretamente.

[master] stop redis master: O quórum é capaz de eleger um novo master com sucesso. Faça o log abaixo.

==> /tmp/sentinel.log <==
[14701] 29 Dec 18:16:55.096 # +sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:16:55.096 # +sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14701] 29 Dec 18:18:04.187 # -sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:18:04.236 # -sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:18:14.160 * +convert-to-slave slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:47:47.151 # +sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14701] 29 Dec 18:47:47.170 # +sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14701] 29 Dec 18:47:57.650 * +reboot slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:47:57.652 * +reboot slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:47:57.715 # -sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14701] 29 Dec 18:47:57.738 # -sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:48:08.088 # +sdown master master 192.168.56.102 6380
[14701] 29 Dec 18:48:08.180 # +sdown master master 192.168.56.102 6380
[14701] 29 Dec 18:48:08.280 # +odown master master 192.168.56.102 6380 #quorum 2/2
[14701] 29 Dec 18:48:08.280 # +new-epoch 73
[14701] 29 Dec 18:48:08.280 # +try-failover master master 192.168.56.102 6380
[14701] 29 Dec 18:48:08.471 # +vote-for-leader a664b9f61df2b10bbbb5d865b01c599ddd36183c 73
[14701] 29 Dec 18:48:08.472 # 192.168.56.103:26379 voted for 491b32b95c547a8266faf9b04ce6b0c18486236b 73
[14705] 29 Dec 18:48:08.473 # +new-epoch 73
[14705] 29 Dec 18:48:08.475 # +vote-for-leader 491b32b95c547a8266faf9b04ce6b0c18486236b 73
[14701] 29 Dec 18:48:08.475 # 192.168.56.102:26380 voted for 491b32b95c547a8266faf9b04ce6b0c18486236b 73
[14701] 29 Dec 18:48:08.835 # +config-update-from sentinel 192.168.56.103:26379 192.168.56.103 26379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:48:08.835 # +config-update-from sentinel 192.168.56.103:26379 192.168.56.103 26379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:48:08.836 # +switch-master master 192.168.56.102 6380 192.168.56.102 6379
[14701] 29 Dec 18:48:08.836 # +switch-master master 192.168.56.102 6380 192.168.56.102 6379
[14701] 29 Dec 18:48:08.836 * +slave slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379
[14705] 29 Dec 18:48:08.836 * +slave slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379

==> /tmp/sent.log <==
Failing master: 192.168.56.102:6380
New Master: 192.168.56.102:6379

Failing master: 192.168.56.102:6380
New Master: 192.168.56.102:6379


==> /tmp/sentinel.log <==
[14705] 29 Dec 18:48:11.855 # +sdown slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379
[14701] 29 Dec 18:48:11.903 # +sdown slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379

Você também pode ver que meu script de reconfiguração é chamado com sucesso e gera alguns detalhes na tela.

O problema é quando estou tentando parar a instância do mestre na máquina [rescue]. A sentinela reporta continuamente "-failover-abort-não-eleito mestre resque 192.168.56.103 6379"

==> /tmp/sentinel.log <==
[14705] 29 Dec 18:48:11.855 # +sdown slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379
[14701] 29 Dec 18:48:11.903 # +sdown slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379
[14705] 29 Dec 18:48:43.401 # -sdown slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379
[14701] 29 Dec 18:48:43.433 # -sdown slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379
[14705] 29 Dec 18:48:53.344 * +convert-to-slave slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379
[14705] 29 Dec 18:49:23.617 # +sdown master resque 192.168.56.103 6379
[14701] 29 Dec 18:49:23.625 # +sdown master resque 192.168.56.103 6379
[14705] 29 Dec 18:49:23.674 # +odown master resque 192.168.56.103 6379 #quorum 2/2
[14705] 29 Dec 18:49:23.674 # +new-epoch 74
[14705] 29 Dec 18:49:23.674 # +try-failover master resque 192.168.56.103 6379
[14701] 29 Dec 18:49:23.727 # +odown master resque 192.168.56.103 6379 #quorum 3/2
[14701] 29 Dec 18:49:23.727 # +new-epoch 74
[14701] 29 Dec 18:49:23.727 # +try-failover master resque 192.168.56.103 6379
[14705] 29 Dec 18:49:23.886 # +vote-for-leader b608fcab7a201799826f4d9ee839aed3cf556fdf 74
[14701] 29 Dec 18:49:23.889 # +vote-for-leader a664b9f61df2b10bbbb5d865b01c599ddd36183c 74
[14701] 29 Dec 18:49:23.890 # 192.168.56.102:26380 voted for b608fcab7a201799826f4d9ee839aed3cf556fdf 74
[14705] 29 Dec 18:49:23.890 # 192.168.56.102:26379 voted for a664b9f61df2b10bbbb5d865b01c599ddd36183c 74
[14705] 29 Dec 18:49:23.893 # 192.168.56.103:26379 voted for b608fcab7a201799826f4d9ee839aed3cf556fdf 74
[14701] 29 Dec 18:49:23.893 # 192.168.56.103:26379 voted for b608fcab7a201799826f4d9ee839aed3cf556fdf 74
[14705] 29 Dec 18:49:23.980 # +elected-leader master resque 192.168.56.103 6379
[14705] 29 Dec 18:49:23.980 # +failover-state-select-slave master resque 192.168.56.103 6379
[14705] 29 Dec 18:49:24.064 # -failover-abort-no-good-slave master resque 192.168.56.103 6379
[14705] 29 Dec 18:49:24.117 # Next failover delay: I will not start a failover before Mon Dec 29 18:49:32 2014
[14701] 29 Dec 18:49:28.417 # -failover-abort-not-elected master resque 192.168.56.103 6379
[14701] 29 Dec 18:49:28.489 # Next failover delay: I will not start a failover before Mon Dec 29 18:49:32 2014
[14705] 29 Dec 18:49:32.217 # +new-epoch 75
[14705] 29 Dec 18:49:32.217 # +try-failover master resque 192.168.56.103 6379
[14701] 29 Dec 18:49:32.423 # +new-epoch 75
[14701] 29 Dec 18:49:32.424 # +try-failover master resque 192.168.56.103 6379
[14705] 29 Dec 18:49:32.433 # +vote-for-leader b608fcab7a201799826f4d9ee839aed3cf556fdf 75
[14705] 29 Dec 18:49:32.435 # 192.168.56.103:26379 voted for 491b32b95c547a8266faf9b04ce6b0c18486236b 75
[14701] 29 Dec 18:49:32.437 # +vote-for-leader a664b9f61df2b10bbbb5d865b01c599ddd36183c 75
[14701] 29 Dec 18:49:32.438 # 192.168.56.102:26380 voted for b608fcab7a201799826f4d9ee839aed3cf556fdf 75
[14705] 29 Dec 18:49:32.438 # 192.168.56.102:26379 voted for a664b9f61df2b10bbbb5d865b01c599ddd36183c 75
[14701] 29 Dec 18:49:32.438 # 192.168.56.103:26379 voted for 491b32b95c547a8266faf9b04ce6b0c18486236b 75
[14705] 29 Dec 18:49:36.636 # -failover-abort-not-elected master resque 192.168.56.103 6379
[14705] 29 Dec 18:49:36.691 # Next failover delay: I will not start a failover before Mon Dec 29 18:49:40 2014
[14701] 29 Dec 18:49:36.843 # -failover-abort-not-elected master resque 192.168.56.103 6379
[14701] 29 Dec 18:49:36.905 # Next failover delay: I will not start a failover before Mon Dec 29 18:49:40 2014

Um novo mestre não é eleito e o script de reconfiguração não é chamado.

Eu entendo que para um novo mestre ser eleito, o quorum (n / 2 + 1) terá que concordar. É por isso que estou usando 3 sentinelas para testar.

Eu não entendo porque o exemplo acima não termina em uma eleição (como [master]).

Estou usando o servidor Redis v = 2.8.19 sha = 0a21368c: 1 malloc = jemalloc-3.6.0 bits = 64 build = e570b291804f6e35

Obrigado por qualquer ajuda e, por favor, não se importe com o grave erro ortográfico da palavra resgate!

- editar

[master] redis slave config:

daemonize yes
pidfile "/var/run/redis/redis-server-slave.pid"
port 6380
tcp-backlog 511
bind 192.168.56.102
timeout 0
tcp-keepalive 0
loglevel notice
logfile "/var/log/redis/redis-server.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
dir "/var/lib/redis"
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
maxclients 4064
# Generated by CONFIG REWRITE

slaveof 192.168.56.102 6379

instância master [master] run_id: 5c1ffb7742ad78cde12dbe4858747a314adaebe9

instância mestre do escravo run_id: 5772f437519bb38782d38f6675ae5d9157be2419

instância principal do [rescue] run_id: 5c1ffb7742ad78cde12dbe4858747a314adaebe9

[rescue] instância escrava run_id: 5772f437519bb38782d38f6675ae5d9157be2419

-

[master] Sentinel (: 26379) run_id: a664b9f61df2b10bbbb5d865b01c599ddd36183c

[master] Sentinel (: 26380) run_id: b608fcab7a201799826f4d9ee839aed3cf556fdf

[escravo] Sentinel (: 26379) run_id: a664b9f61df2b10bbbb5d865b01c599ddd36183c

Informações do Sentinel

# Sentinel
sentinel_masters:2
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
master0:name=resque,status=ok,address=192.168.56.103:6379,slaves=0,sentinels=3
master1:name=master,status=ok,address=192.168.56.102:6379,slaves=1,sentinels=3

Sentinel conf de 192.168.56.103

port 26379
logfile "/tmp/sentinel.log"
dir "/tmp"

sentinel monitor resque 192.168.56.103 6379 2
sentinel down-after-milliseconds resque 3000
sentinel failover-timeout resque 4000
sentinel client-reconfig-script resque /home/sm0ke/Projects/git/thrace/extra/sentinel-failover.py

sentinel config-epoch resque 0
sentinel leader-epoch resque 76
sentinel known-sentinel resque 192.168.56.102 26379 a664b9f61df2b10bbbb5d865b01c599ddd36183c
sentinel known-sentinel resque 192.168.56.102 26380 b608fcab7a201799826f4d9ee839aed3cf556fdf

maxclients 4064
daemonize yes
# Generated by CONFIG REWRITE
sentinel monitor master 192.168.56.102 6379 2
sentinel down-after-milliseconds master 3000
sentinel failover-timeout master 4000
sentinel client-reconfig-script master /home/sm0ke/Projects/git/thrace/extra/sentinel-failover.py
sentinel config-epoch master 73
sentinel leader-epoch master 73
sentinel known-slave master 192.168.56.102 6380
sentinel known-sentinel master 192.168.56.102 26380 b608fcab7a201799826f4d9ee839aed3cf556fdf
sentinel known-sentinel master 192.168.56.102 26379 a664b9f61df2b10bbbb5d865b01c599ddd36183c
sentinel current-epoch 76

informações do redis-cli para o Sentinel em 192.168.56.103

# Server
redis_version:2.8.19
redis_git_sha1:0a21368c
redis_git_dirty:1
redis_build_id:e570b291804f6e35
redis_mode:sentinel
os:Linux 3.2.0-4-amd64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.7.2
process_id:8345
run_id:491b32b95c547a8266faf9b04ce6b0c18486236b
tcp_port:26379
uptime_in_seconds:8054
uptime_in_days:0
hz:17
lru_clock:10594345
config_file:/etc/redis/sentinel.conf

# Sentinel
sentinel_masters:2
sentinel_tilt:0
sentinel_running_scripts:16
sentinel_scripts_queue_length:4
master0:name=resque,status=ok,address=192.168.56.103:6379,slaves=0,sentinels=3
master1:name=master,status=ok,address=192.168.56.102:6379,slaves=1,sentinels=3

redis cluster failover

por sm0ke21 29.12.2014 / 19:10

1 resposta

Tags redis cluster failover

Substituindo uma unidade com falha - RAID 1 Atualização do AWS RDS Postgres sem perder dados

score 2 · Accepted Answer

Ignore-me sobre a confusão de ID de instância, a culpa é minha - esqueci que os sentinelas estavam votando em um líder entre os sentinelas, não votando em um novo mestre entre os candidatos, então os três IDs diferentes fazem sentido.

Então, aqui está o problema real:

master0:name=resque,status=ok,address=192.168.56.103:6379,slaves=0,sentinels=3

Não há escravos que as sentinelas tenham notado, então eles não têm um bom nó para fazer ao vivo durante a votação.

Verifique a slaveof config na instância 192.168.56.103:6380 , certifique-se de que a instância está em execução e conecte-se a ela para verificar sua info para verificar se ela está funcionando. Uma vez que o 192.168.56.103:6379 o veja como um escravo (em seu comando info ), as sentinelas o pegarão como um escravo conhecido e serão capazes de fazer failover nele.