三叶运维redis 哨兵(Sentinel)
redis 集群介绍
主从架构无法实现master和slave角色的自动切换，即当master出现redis服务异常、主机断电、磁盘损坏等问题导致master无法使用，而redis高可用无法实现自故障转移(将slave提升为master)，需要手动改环境配置才能切换到slave redis服务器，另外也无法横向扩展Redis服务的并行写入性能，当单台Redis服务器性能无法满足业务写入需求的时候就必须需要一种方式解决以上的两个核心问题，即：1.master和slave角色的无缝切换，让业务无感知从而不影响业务使用 2.可以横向动态扩展Redis服务器，从而实现多台服务器并行写入以实现更高并发的目的。
Redis 集群实现方式：
- 客户端分片
- 代理分片
- Redis Cluster
哨兵(Sentinel) 工作原理
Sentinel 进程是用于监控redis集群中Master主服务器工作的状态，在Master主服务器发生故障的时候，可以实现
Master和Slave服务器的切换，保证系统的高可用，其已经被集成在redis2.6+的版本中，Redis的哨兵模式到了2.8
版本之后就稳定了下来。一般在生产环境也建议使用Redis的2.8版本的以后版本。哨兵(Sentinel) 是一个分布式系
统，可以在一个架构中运行多个哨兵(sentinel) 进程，这些进程使用流言协议(gossip protocols)来接收关于Master
主服务器是否下线的信息，并使用投票协议(Agreement Protocols)来决定是否执行自动故障迁移,以及选择哪个
Slave作为新的Master。
每个哨兵(Sentinel)进程会向其它哨兵(Sentinel)、Master、Slave定时发送消息，以确认对方是否”活”着，如果发现对方在指定配置时间(可配置的)内未得到回应，则暂时认为对方已离线，也就是所谓的”主观认为宕机” ，主观是每个成员都具有的独自的而且可能相同也可能不同的意识，英文名称：Subjective Down，简称SDOWN。
有主观宕机，对应的就有客观宕机。当“哨兵群”中的多数Sentinel进程在对Master主服务器做出SDOWN 的判断，并且通过 SENTINEL is-master-down-by-addr 命令互相交流之后，得出的Master Server下线判断，这种方式就是“客观宕机”，客观是不依赖于某种意识而已经实际存在的一切事物，英文名称是：Objectively Down， 简称 ODOWN。
通过一定的vote算法，从剩下的slave从服务器节点中，选一台提升为Master服务器节点，然后自动修改相关配置，并开启故障转移（failover）。Sentinel 机制可以解决master和slave角色的自动切换问题，但单个Master 的性能瓶颈问题无法解决
实现哨兵
哨兵的准备
哨兵的前提是已经实现了一个redis master-slave的运行环境，从而实现一个一主两从基于哨兵的高可用redis架构
master服务器状态
[root@redis-master ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not 127.0.0.1:6379> INFO replication # Replication role:master connected_slaves:2 slave0:ip=10.0.0.28,port=6379,state=online,offset=112,lag=1 slave1:ip=10.0.0.18,port=6379,state=online,offset=112,lag=0 master_replid:8fdca730a2ae48fb9c8b7e739dcd2efcc76794f3 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:112 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:112 127.0.0.1:6379>
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | [ root @ redis - master ~ ] #redis-cli -a 123456 Warning : Using a password with '-a' or '-u' option on the command line interface may not 127.0.0.1 : 6379 > INFO replication # Replication role : master connected_slaves : 2 slave0 : ip = 10.0.0.28 , port = 6379 , state = online , offset = 112 , lag = 1 slave1 : ip = 10.0.0.18 , port = 6379 , state = online , offset = 112 , lag = 0 master_replid : 8fdca730a2ae48fb9c8b7e739dcd2efcc76794f3 master_replid2 : 0000000000000000000000000000000000000000 master_repl_offset : 112 second_repl_offset : - 1 repl_backlog_active : 1 repl_backlog_size : 1048576 repl_backlog_first_byte_offset : 1 repl_backlog_histlen : 112 127.0.0.1 : 6379 > |
配置slave1
[root@redis-slave1 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> REPLICAOF 10.0.0.8 6379 OK 127.0.0.1:6379> CONFIG SET masterauth "123456" OK 127.0.0.1:6379> INFO replication # Replication role:slave master_host:10.0.0.8 master_port:6379 master_link_status:up master_last_io_seconds_ago:4 master_sync_in_progress:0 slave_repl_offset:140 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:8fdca730a2ae48fb9c8b7e739dcd2efcc76794f3 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:140 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:99 repl_backlog_histlen:42
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | [ root @ redis - slave1 ~ ] #redis-cli -a 123456 Warning : Using a password with '-a' or '-u' option on the command line interface may not be safe . 127.0.0.1 : 6379 > REPLICAOF 10.0.0.8 6379 OK 127.0.0.1 : 6379 > CONFIG SET masterauth "123456" OK 127.0.0.1 : 6379 > INFO replication # Replication role : slave master_host : 10.0.0.8 master_port : 6379 master_link_status : up master_last_io_seconds_ago : 4 master_sync_in_progress : 0 slave_repl_offset : 140 slave_priority : 100 slave_read_only : 1 connected_slaves : 0 master_replid : 8fdca730a2ae48fb9c8b7e739dcd2efcc76794f3 master_replid2 : 0000000000000000000000000000000000000000 master_repl_offset : 140 second_repl_offset : - 1 repl_backlog_active : 1 repl_backlog_size : 1048576 repl_backlog_first_byte_offset : 99 repl_backlog_histlen : 42 |
配置slave2
[root@redis-slave2 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> REPLICAOF 10.0.0.8 6379 OK 127.0.0.1:6379> CONFIG SET masterauth "123456" OK 127.0.0.1:6379> INFO replication # Replication role:slave master_host:10.0.0.8 master_port:6379 master_link_status:up master_last_io_seconds_ago:3 master_sync_in_progress:0 slave_repl_offset:182 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:8fdca730a2ae48fb9c8b7e739dcd2efcc76794f3 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:182 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:15 repl_backlog_histlen:168 127.0.0.1:6379>
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | [ root @ redis - slave2 ~ ] #redis-cli -a 123456 Warning : Using a password with '-a' or '-u' option on the command line interface may not be safe . 127.0.0.1 : 6379 > REPLICAOF 10.0.0.8 6379 OK 127.0.0.1 : 6379 > CONFIG SET masterauth "123456" OK 127.0.0.1 : 6379 > INFO replication # Replication role : slave master_host : 10.0.0.8 master_port : 6379 master_link_status : up master_last_io_seconds_ago : 3 master_sync_in_progress : 0 slave_repl_offset : 182 slave_priority : 100 slave_read_only : 1 connected_slaves : 0 master_replid : 8fdca730a2ae48fb9c8b7e739dcd2efcc76794f3 master_replid2 : 0000000000000000000000000000000000000000 master_repl_offset : 182 second_repl_offset : - 1 repl_backlog_active : 1 repl_backlog_size : 1048576 repl_backlog_first_byte_offset : 15 repl_backlog_histlen : 168 127.0.0.1 : 6379 > |
编辑配置文件sentinel.conf
sentinel.conf的配置
哨兵可不和Redis服务器部署在一起，但一般部署在一起，所有redis节点使用相同的以下示例的配置文件
#如果是编译安装，在源码目录有sentinel.conf，复制到安装目录即可，如:/apps/redis/etc/sentinel.conf [root@centos8 ~]#vim /etc/redis-sentinel.conf bind 0.0.0.0 port 26379 daemonize yes pidfile "redis-sentinel.pid" logfile "sentinel_26379.log" dir "/tmp" #工作目录 sentinel myid 50547f34ed71fd48c197924969937e738a39975b sentinel monitor mymaster 10.0.0.100 6379 2 #指定master服务器的地址和端口 #2为法定人数限制(quorum)，即有几个slave认为master down了就进行故障转移，一般此值是所有节点的一半以上的整数值，比如，总数是3，即3/2=1.5，取整为2 sentinel auth-pass mymaster 123456 #master的密码，注意此行要在上面行的下面 sentinel down-after-milliseconds mymaster 30000 #(SDOWN)主观下线的时间，单位：毫秒，建议3000 sentinel parallel-syncs mymaster 1 #发生故障转移后，同时向新master同步数据的slave数量，数字越小总同步时间越长，但可以减轻新master的负载压力 sentinel failover-timeout mymaster 180000 #所有slaves指向新的master所需的超时时间，单位：毫秒 sentinel deny-scripts-reconfig yes #禁止修改脚本 logfile /var/log/redis/sentinel.log
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | #如果是编译安装，在源码目录有sentinel.conf，复制到安装目录即可，如:/apps/redis/etc/sentinel.conf [ root @ centos8 ~ ] #vim /etc/redis-sentinel.conf bind 0.0.0.0 port 26379 daemonize yes pidfile "redis-sentinel.pid" logfile "sentinel_26379.log" dir "/tmp" #工作目录 sentinel myid 50547f34ed71fd48c197924969937e738a39975b sentinel monitor mymaster 10.0.0.100 6379 2 #指定master服务器的地址和端口 #2为法定人数限制(quorum)，即有几个slave认为master down了就进行故障转移，一般此值是所有节点的一半以上的整数值，比如，总数是3，即3/2=1.5，取整为2 sentinel auth - pass mymaster 123456 #master的密码，注意此行要在上面行的下面 sentinel down - after - milliseconds mymaster 30000 #(SDOWN)主观下线的时间，单位：毫秒，建议3000 sentinel parallel - syncs mymaster 1 #发生故障转移后，同时向新master同步数据的slave数量，数字越小总同步时间越长，但可以减轻新master的负载压力 sentinel failover - timeout mymaster 180000 #所有slaves指向新的master所需的超时时间，单位：毫秒 sentinel deny - scripts - reconfig yes #禁止修改脚本 logfile / var / log / redis / sentinel . log |
三个哨后服务器的配置都如下
[root@redis-master ~]#grep -vE "^#|^$" /etc/redis-sentinel.conf port 26379 daemonize no pidfile "/var/run/redis-sentinel.pid" logfile "/var/log/redis/sentinel.log" dir "/tmp" sentinel myid 50547f34ed71fd48c197924969937e738a39975b #此行每个哨兵主机必须唯一 sentinel deny-scripts-reconfig yes sentinel monitor mymaster 10.0.0.8 6379 2 #修改此行 sentinel down-after-milliseconds mymaster 3000 #修改此行 sentinel auth-pass mymaster 123456 #增加此行 sentinel config-epoch mymaster 0 #以下自动生成，不需要修改 protected-mode no supervised systemd sentinel leader-epoch mymaster 0 sentinel known-replica mymaster 10.0.0.28 6379 sentinel known-replica mymaster 10.0.0.18 6379 sentinel current-epoch 0 [root@redis-master ~]#scp /etc/redis-sentinel.conf redis-slave1:/etc/ [root@redis-master ~]#scp /etc/redis-sentinel.conf redis-slave2:/etc/ sentinel myid 50547f34ed71fd48c197924969937e738a39975b
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | [ root @ redis - master ~ ] #grep -vE "^#|^$" /etc/redis-sentinel.conf port 26379 daemonize no pidfile "/var/run/redis-sentinel.pid" logfile "/var/log/redis/sentinel.log" dir "/tmp" sentinel myid 50547f34ed71fd48c197924969937e738a39975b #此行每个哨兵主机必须唯一 sentinel deny - scripts - reconfig yes sentinel monitor mymaster 10.0.0.8 6379 2 #修改此行 sentinel down - after - milliseconds mymaster 3000 #修改此行 sentinel auth - pass mymaster 123456 #增加此行 sentinel config - epoch mymaster 0 #以下自动生成，不需要修改 protected - mode no supervised systemd sentinel leader - epoch mymaster 0 sentinel known - replica mymaster 10.0.0.28 6379 sentinel known - replica mymaster 10.0.0.18 6379 sentinel current - epoch 0 [ root @ redis - master ~ ] #scp /etc/redis-sentinel.conf redis-slave1:/etc/ [ root @ redis - master ~ ] #scp /etc/redis-sentinel.conf redis-slave2:/etc/ sentinel myid 50547f34ed71fd48c197924969937e738a39975b |
启动哨兵
三台哨兵服务器都要启动
#确保每个哨兵主机myid不同 [root@redis-slave1 ~]#vim /etc/redis-sentinel.conf sentinel myid 50547f34ed71fd48c197924969937e738a39975c [root@redis-slave2 ~]#vim /etc/redis-sentinel.conf sentinel myid 50547f34ed71fd48c197924969937e738a39975d [root@redis-master ~]#systemctl restart redis-sentinel.service [root@redis-slave1 ~]#systemctl restart redis-sentinel.service [root@redis-slave2 ~]#systemctl restart redis-sentinel.service
| 1 2 3 4 5 6 7 8 9 | #确保每个哨兵主机myid不同 [ root @ redis - slave1 ~ ] #vim /etc/redis-sentinel.conf sentinel myid 50547f34ed71fd48c197924969937e738a39975c [ root @ redis - slave2 ~ ] #vim /etc/redis-sentinel.conf sentinel myid 50547f34ed71fd48c197924969937e738a39975d [ root @ redis - master ~ ] #systemctl restart redis-sentinel.service [ root @ redis - slave1 ~ ] #systemctl restart redis-sentinel.service [ root @ redis - slave2 ~ ] #systemctl restart redis-sentinel.service |
如果是编译安装执行下面类似操作
#/apps/redis/bin/redis-sentinel /apps/redis/etc/sentinel.conf
| 1 | #/apps/redis/bin/redis-sentinel /apps/redis/etc/sentinel.conf |
验证哨兵端口
[root@redis-master ~]#ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 0.0.0.0:26379 0.0.0.0:* LISTEN 0 128 0.0.0.0:6379 0.0.0.0:* LISTEN 0 128 [::]:22 [::]:* LISTEN 0 128 [::]:26379 [::]:* LISTEN 0 128 [::]:6379 [::]:*
| 1 2 3 4 5 6 7 8 | [ root @ redis - master ~ ] #ss -ntl State Recv - Q Send - Q Local Address : Port Peer Address : Port LISTEN 0 128 0.0.0.0 : 22 0.0.0.0 : * LISTEN 0 128 0.0.0.0 : 26379 0.0.0.0 : * LISTEN 0 128 0.0.0.0 : 6379 0.0.0.0 : * LISTEN 0 128 [ :: ] : 22 [ :: ] : * LISTEN 0 128 [ :: ] : 26379 [ :: ] : * LISTEN 0 128 [ :: ] : 6379 [ :: ] : * |
查看哨兵日志
master的哨兵日志
[root@redis-master ~]#tail -f /var/log/redis/sentinel.log 38028:X 20 Feb 2020 17:13:08.702 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 38028:X 20 Feb 2020 17:13:08.702 # Redis version=5.0.3, bits=64, commit=00000000, modified=0, pid=38028, just started 38028:X 20 Feb 2020 17:13:08.702 # Configuration loaded 38028:X 20 Feb 2020 17:13:08.702 * supervised by systemd, will signal readiness 38028:X 20 Feb 2020 17:13:08.703 * Running mode=sentinel, port=26379. 38028:X 20 Feb 2020 17:13:08.703 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 38028:X 20 Feb 2020 17:13:08.704 # Sentinel ID is 50547f34ed71fd48c197924969937e738a39975b 38028:X 20 Feb 2020 17:13:08.704 # +monitor master mymaster 10.0.0.8 6379 quorum 2 38028:X 20 Feb 2020 17:13:08.709 * +slave slave 10.0.0.28:6379 10.0.0.28 6379 @ mymaster 10.0.0.8 6379 38028:X 20 Feb 2020 17:13:08.709 * +slave slave 10.0.0.18:6379 10.0.0.18 6379 @ mymaster 10.0.0.8 6379
| 1 2 3 4 5 6 7 8 9 10 11 | [ root @ redis - master ~ ] #tail -f /var/log/redis/sentinel.log 38028 : X 20 Feb 2020 17 : 13 : 08.702 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 38028 : X 20 Feb 2020 17 : 13 : 08.702 # Redis version=5.0.3, bits=64, commit=00000000, modified=0, pid=38028, just started 38028 : X 20 Feb 2020 17 : 13 : 08.702 # Configuration loaded 38028 : X 20 Feb 2020 17 : 13 : 08.702 * supervised by systemd , will signal readiness 38028 : X 20 Feb 2020 17 : 13 : 08.703 * Running mode = sentinel , port = 26379. 38028 : X 20 Feb 2020 17 : 13 : 08.703 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 38028 : X 20 Feb 2020 17 : 13 : 08.704 # Sentinel ID is 50547f34ed71fd48c197924969937e738a39975b 38028 : X 20 Feb 2020 17 : 13 : 08.704 # +monitor master mymaster 10.0.0.8 6379 quorum 2 38028 : X 20 Feb 2020 17 : 13 : 08.709 * + slave slave 10.0.0.28 : 6379 10.0.0.28 6379 @ mymaster 10.0.0.8 6379 38028 : X 20 Feb 2020 17 : 13 : 08.709 * + slave slave 10.0.0.18 : 6379 10.0.0.18 6379 @ mymaster 10.0.0.8 6379 |
slave的哨兵日志
[root@redis-slave1 ~]#tail -f /var/log/redis/sentinel.log 25509:X 20 Feb 2020 17:13:27.435 * Removing the pid file. 25509:X 20 Feb 2020 17:13:27.435 # Sentinel is now ready to exit, bye bye... 25572:X 20 Feb 2020 17:13:27.448 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 25572:X 20 Feb 2020 17:13:27.448 # Redis version=5.0.3, bits=64, commit=00000000, modified=0, pid=25572, just started 25572:X 20 Feb 2020 17:13:27.448 # Configuration loaded 25572:X 20 Feb 2020 17:13:27.448 * supervised by systemd, will signal readiness 25572:X 20 Feb 2020 17:13:27.449 * Running mode=sentinel, port=26379. 25572:X 20 Feb 2020 17:13:27.449 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 25572:X 20 Feb 2020 17:13:27.449 # Sentinel ID is 50547f34ed71fd48c197924969937e738a39975b 25572:X 20 Feb 2020 17:13:27.449 # +monitor master mymaster 10.0.0.8 6379 quorum 2
| 1 2 3 4 5 6 7 8 9 10 11 | [ root @ redis - slave1 ~ ] #tail -f /var/log/redis/sentinel.log 25509 : X 20 Feb 2020 17 : 13 : 27.435 * Removing the pid file . 25509 : X 20 Feb 2020 17 : 13 : 27.435 # Sentinel is now ready to exit, bye bye... 25572 : X 20 Feb 2020 17 : 13 : 27.448 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 25572 : X 20 Feb 2020 17 : 13 : 27.448 # Redis version=5.0.3, bits=64, commit=00000000, modified=0, pid=25572, just started 25572 : X 20 Feb 2020 17 : 13 : 27.448 # Configuration loaded 25572 : X 20 Feb 2020 17 : 13 : 27.448 * supervised by systemd , will signal readiness 25572 : X 20 Feb 2020 17 : 13 : 27.449 * Running mode = sentinel , port = 26379. 25572 : X 20 Feb 2020 17 : 13 : 27.449 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 25572 : X 20 Feb 2020 17 : 13 : 27.449 # Sentinel ID is 50547f34ed71fd48c197924969937e738a39975b 25572 : X 20 Feb 2020 17 : 13 : 27.449 # +monitor master mymaster 10.0.0.8 6379 quorum 2 |
当前sentinel状态
在sentinel状态中尤其是最后一行，涉及到masterIP是多少，有几个slave，有几个sentinels，必须是符合全部服
务器数量的。
[root@redis-master ~]#redis-cli -p 26379 127.0.0.1:26379> INFO sentinel # Sentinel sentinel_masters:1 sentinel_tilt:0 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 sentinel_simulate_failure_flags:0 master0:name=mymaster,status=ok,address=10.0.0.8:6379,slaves=2,sentinels=1
| 1 2 3 4 5 6 7 8 9 | [ root @ redis - master ~ ] #redis-cli -p 26379 127.0.0.1 : 26379 > INFO sentinel # Sentinel sentinel_masters : 1 sentinel_tilt : 0 sentinel_running_scripts : 0 sentinel_scripts_queue_length : 0 sentinel_simulate_failure_flags : 0 master0 : name = mymaster , status = ok , address = 10.0.0.8 : 6379 , slaves = 2 , sentinels = 1 |
停止Redis Master测试故障转移
[root@redis-master ~]#killall redis-server
| 1 | [ root @ redis - master ~ ] #killall redis-server |
查看各节点上哨兵信息：
[root@redis-master ~]#redis-cli -a 123456 -p 26379 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:26379> INFO sentinel # Sentinel sentinel_masters:1 sentinel_tilt:0 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 sentinel_simulate_failure_flags:0 master0:name=mymaster,status=ok,address=10.0.0.18:6379,slaves=2,sentinels=2
| 1 2 3 4 5 6 7 8 9 10 11 | [ root @ redis - master ~ ] #redis-cli -a 123456 -p 26379 Warning : Using a password with '-a' or '-u' option on the command line interface may not be safe . 127.0.0.1 : 26379 > INFO sentinel # Sentinel sentinel_masters : 1 sentinel_tilt : 0 sentinel_running_scripts : 0 sentinel_scripts_queue_length : 0 sentinel_simulate_failure_flags : 0 master0 : name = mymaster , status = ok , address = 10.0.0.18 : 6379 , slaves = 2 , sentinels = 2 |
故障转移时sentinel的信息：
[root@redis-master ~]#tail -f /var/log/redis/sentinel.log 38028:X 20 Feb 2020 17:42:27.362 # +sdown master mymaster 10.0.0.8 6379 38028:X 20 Feb 2020 17:42:27.418 # +odown master mymaster 10.0.0.8 6379 #quorum 2/2 38028:X 20 Feb 2020 17:42:27.418 # +new-epoch 1 38028:X 20 Feb 2020 17:42:27.418 # +try-failover master mymaster 10.0.0.8 6379 38028:X 20 Feb 2020 17:42:27.419 # +vote-for-leader 50547f34ed71fd48c197924969937e738a39975b 1 38028:X 20 Feb 2020 17:42:27.422 # 50547f34ed71fd48c197924969937e738a39975d voted for 50547f34ed71fd48c197924969937e738a39975b 1 38028:X 20 Feb 2020 17:42:27.475 # +elected-leader master mymaster 10.0.0.8 6379 38028:X 20 Feb 2020 17:42:27.475 # +failover-state-select-slave master mymaster 10.0.0.8 6379 38028:X 20 Feb 2020 17:42:27.529 # +selected-slave slave 10.0.0.18:6379 10.0.0.18 6379 @ mymaster 10.0.0.8 6379 38028:X 20 Feb 2020 17:42:27.529 * +failover-state-send-slaveof-noone slave 10.0.0.18:6379 10.0.0.18 6379 @ mymaster 10.0.0.8 6379 38028:X 20 Feb 2020 17:42:27.613 * +failover-state-wait-promotion slave 10.0.0.18:6379 10.0.0.18 6379 @ mymaster 10.0.0.8 6379 38028:X 20 Feb 2020 17:42:28.506 # +promoted-slave slave 10.0.0.18:6379 10.0.0.18 6379 @ mymaster 10.0.0.8 6379 38028:X 20 Feb 2020 17:42:28.506 # +failover-state-reconf-slaves master mymaster 10.0.0.8 6379 38028:X 20 Feb 2020 17:42:28.582 * +slave-reconf-sent slave 10.0.0.28:6379 10.0.0.28 6379 @ mymaster 10.0.0.8 6379 38028:X 20 Feb 2020 17:42:28.736 * +slave-reconf-inprog slave 10.0.0.28:6379 10.0.0.28 6379 @ mymaster 10.0.0.8 6379 38028:X 20 Feb 2020 17:42:28.736 * +slave-reconf-done slave 10.0.0.28:6379 10.0.0.28 6379 @ mymaster 10.0.0.8 6379 38028:X 20 Feb 2020 17:42:28.799 # +failover-end master mymaster 10.0.0.8 6379 38028:X 20 Feb 2020 17:42:28.799 # +switch-master mymaster 10.0.0.8 6379 10.0.0.18 6379 38028:X 20 Feb 2020 17:42:28.799 * +slave slave 10.0.0.28:6379 10.0.0.28 6379 @ mymaster 10.0.0.18 6379 38028:X 20 Feb 2020 17:42:28.799 * +slave slave 10.0.0.8:6379 10.0.0.8 6379 @ mymaster 10.0.0.18 6379 38028:X 20 Feb 2020 17:42:31.809 # +sdown slave 10.0.0.8:6379 10.0.0.8 6379 @ mymaster 10.0.0.18 6379
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | [ root @ redis - master ~ ] #tail -f /var/log/redis/sentinel.log 38028 : X 20 Feb 2020 17 : 42 : 27.362 # +sdown master mymaster 10.0.0.8 6379 38028 : X 20 Feb 2020 17 : 42 : 27.418 # +odown master mymaster 10.0.0.8 6379 #quorum 2/2 38028 : X 20 Feb 2020 17 : 42 : 27.418 # +new-epoch 1 38028 : X 20 Feb 2020 17 : 42 : 27.418 # +try-failover master mymaster 10.0.0.8 6379 38028 : X 20 Feb 2020 17 : 42 : 27.419 # +vote-for-leader 50547f34ed71fd48c197924969937e738a39975b 1 38028 : X 20 Feb 2020 17 : 42 : 27.422 # 50547f34ed71fd48c197924969937e738a39975d voted for 50547f34ed71fd48c197924969937e738a39975b 1 38028 : X 20 Feb 2020 17 : 42 : 27.475 # +elected-leader master mymaster 10.0.0.8 6379 38028 : X 20 Feb 2020 17 : 42 : 27.475 # +failover-state-select-slave master mymaster 10.0.0.8 6379 38028 : X 20 Feb 2020 17 : 42 : 27.529 # +selected-slave slave 10.0.0.18:6379 10.0.0.18 6379 @ mymaster 10.0.0.8 6379 38028 : X 20 Feb 2020 17 : 42 : 27.529 * + failover - state - send - slaveof - noone slave 10.0.0.18 : 6379 10.0.0.18 6379 @ mymaster 10.0.0.8 6379 38028 : X 20 Feb 2020 17 : 42 : 27.613 * + failover - state - wait - promotion slave 10.0.0.18 : 6379 10.0.0.18 6379 @ mymaster 10.0.0.8 6379 38028 : X 20 Feb 2020 17 : 42 : 28.506 # +promoted-slave slave 10.0.0.18:6379 10.0.0.18 6379 @ mymaster 10.0.0.8 6379 38028 : X 20 Feb 2020 17 : 42 : 28.506 # +failover-state-reconf-slaves master mymaster 10.0.0.8 6379 38028 : X 20 Feb 2020 17 : 42 : 28.582 * + slave - reconf - sent slave 10.0.0.28 : 6379 10.0.0.28 6379 @ mymaster 10.0.0.8 6379 38028 : X 20 Feb 2020 17 : 42 : 28.736 * + slave - reconf - inprog slave 10.0.0.28 : 6379 10.0.0.28 6379 @ mymaster 10.0.0.8 6379 38028 : X 20 Feb 2020 17 : 42 : 28.736 * + slave - reconf - done slave 10.0.0.28 : 6379 10.0.0.28 6379 @ mymaster 10.0.0.8 6379 38028 : X 20 Feb 2020 17 : 42 : 28.799 # +failover-end master mymaster 10.0.0.8 6379 38028 : X 20 Feb 2020 17 : 42 : 28.799 # +switch-master mymaster 10.0.0.8 6379 10.0.0.18 6379 38028 : X 20 Feb 2020 17 : 42 : 28.799 * + slave slave 10.0.0.28 : 6379 10.0.0.28 6379 @ mymaster 10.0.0.18 6379 38028 : X 20 Feb 2020 17 : 42 : 28.799 * + slave slave 10.0.0.8 : 6379 10.0.0.8 6379 @ mymaster 10.0.0.18 6379 38028 : X 20 Feb 2020 17 : 42 : 31.809 # +sdown slave 10.0.0.8:6379 10.0.0.8 6379 @ mymaster 10.0.0.18 6379 |
故障转移后的redis配置文件会被自动修改
故障转移后redis.conf中的replicaof行的master IP会被修改，。
[root@redis-slave2 ~]#grep ^replicaof /etc/redis.conf replicaof 10.0.0.18 6379
| 1 2 | [ root @ redis - slave2 ~ ] #grep ^replicaof /etc/redis.conf replicaof 10.0.0.18 6379 |
sentinel.conf中的sentinel monitor IP会被修改
[root@redis-slave1 ~]#grep "^[a-Z]" /etc/redis-sentinel.conf port 26379 daemonize no pidfile "/var/run/redis-sentinel.pid" logfile "/var/log/redis/sentinel.log" dir "/tmp" sentinel myid 50547f34ed71fd48c197924969937e738a39975b sentinel deny-scripts-reconfig yes sentinel monitor mymaster 10.0.0.18 6379 2 #自动修改此行 sentinel down-after-milliseconds mymaster 3000 sentinel auth-pass mymaster 123456 sentinel config-epoch mymaster 1 protected-mode no supervised systemd sentinel leader-epoch mymaster 1 sentinel known-replica mymaster 10.0.0.8 6379 sentinel known-replica mymaster 10.0.0.28 6379 sentinel known-sentinel mymaster 10.0.0.28 26379 50547f34ed71fd48c197924969937e738a39975d sentinel current-epoch 1 [root@redis-slave2 ~]#grep "^[a-Z]" /etc/redis-sentinel.conf port 26379 daemonize no pidfile "/var/run/redis-sentinel.pid" logfile "/var/log/redis/sentinel.log" dir "/tmp" sentinel myid 50547f34ed71fd48c197924969937e738a39975d sentinel deny-scripts-reconfig yes sentinel monitor mymaster 10.0.0.18 6379 2 #自动修改此行 sentinel down-after-milliseconds mymaster 3000 sentinel auth-pass mymaster 123456 sentinel config-epoch mymaster 1 protected-mode no supervised systemd sentinel leader-epoch mymaster 1 sentinel known-replica mymaster 10.0.0.28 6379 sentinel known-replica mymaster 10.0.0.8 6379 sentinel known-sentinel mymaster 10.0.0.8 26379 50547f34ed71fd48c197924969937e738a39975b sentinel current-epoch 1
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | [ root @ redis - slave1 ~ ] #grep "^[a-Z]" /etc/redis-sentinel.conf port 26379 daemonize no pidfile "/var/run/redis-sentinel.pid" logfile "/var/log/redis/sentinel.log" dir "/tmp" sentinel myid 50547f34ed71fd48c197924969937e738a39975b sentinel deny - scripts - reconfig yes sentinel monitor mymaster 10.0.0.18 6379 2 #自动修改此行 sentinel down - after - milliseconds mymaster 3000 sentinel auth - pass mymaster 123456 sentinel config - epoch mymaster 1 protected - mode no supervised systemd sentinel leader - epoch mymaster 1 sentinel known - replica mymaster 10.0.0.8 6379 sentinel known - replica mymaster 10.0.0.28 6379 sentinel known - sentinel mymaster 10.0.0.28 26379 50547f34ed71fd48c197924969937e738a39975d sentinel current - epoch 1 [ root @ redis - slave2 ~ ] #grep "^[a-Z]" /etc/redis-sentinel.conf port 26379 daemonize no pidfile "/var/run/redis-sentinel.pid" logfile "/var/log/redis/sentinel.log" dir "/tmp" sentinel myid 50547f34ed71fd48c197924969937e738a39975d sentinel deny - scripts - reconfig yes sentinel monitor mymaster 10.0.0.18 6379 2 #自动修改此行 sentinel down - after - milliseconds mymaster 3000 sentinel auth - pass mymaster 123456 sentinel config - epoch mymaster 1 protected - mode no supervised systemd sentinel leader - epoch mymaster 1 sentinel known - replica mymaster 10.0.0.28 6379 sentinel known - replica mymaster 10.0.0.8 6379 sentinel known - sentinel mymaster 10.0.0.8 26379 50547f34ed71fd48c197924969937e738a39975b sentinel current - epoch 1 |
当前reids状态
新的master 状态
[root@redis-slave1 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> INFO replication # Replication role:master #提升为master connected_slaves:1 slave0:ip=10.0.0.28,port=6379,state=online,offset=56225,lag=1 master_replid:75e3f205082c5a10824fbe6580b6ad4437140b94 master_replid2:b2fb4653bdf498691e5f88519ded65b6c000e25c master_repl_offset:56490 second_repl_offset:46451 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:287 repl_backlog_histlen:56204
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | [ root @ redis - slave1 ~ ] #redis-cli -a 123456 Warning : Using a password with '-a' or '-u' option on the command line interface may not be safe . 127.0.0.1 : 6379 > INFO replication # Replication role : master #提升为master connected_slaves : 1 slave0 : ip = 10.0.0.28 , port = 6379 , state = online , offset = 56225 , lag = 1 master_replid : 75e3f205082c5a10824fbe6580b6ad4437140b94 master_replid2 : b2fb4653bdf498691e5f88519ded65b6c000e25c master_repl_offset : 56490 second_repl_offset : 46451 repl_backlog_active : 1 repl_backlog_size : 1048576 repl_backlog_first_byte_offset : 287 repl_backlog_histlen : 56204 |
另一个slave指向新的master
[root@redis-slave2 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> INFO replication # Replication role:slave master_host:10.0.0.18 #指向新的master master_port:6379 master_link_status:up master_last_io_seconds_ago:0 master_sync_in_progress:0 slave_repl_offset:61029 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:75e3f205082c5a10824fbe6580b6ad4437140b94 master_replid2:b2fb4653bdf498691e5f88519ded65b6c000e25c master_repl_offset:61029 second_repl_offset:46451 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:61029
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | [ root @ redis - slave2 ~ ] #redis-cli -a 123456 Warning : Using a password with '-a' or '-u' option on the command line interface may not be safe . 127.0.0.1 : 6379 > INFO replication # Replication role : slave master_host : 10.0.0.18 #指向新的master master_port : 6379 master_link_status : up master_last_io_seconds_ago : 0 master_sync_in_progress : 0 slave_repl_offset : 61029 slave_priority : 100 slave_read_only : 1 connected_slaves : 0 master_replid : 75e3f205082c5a10824fbe6580b6ad4437140b94 master_replid2 : b2fb4653bdf498691e5f88519ded65b6c000e25c master_repl_offset : 61029 second_repl_offset : 46451 repl_backlog_active : 1 repl_backlog_size : 1048576 repl_backlog_first_byte_offset : 1 repl_backlog_histlen : 61029 |
恢复故障的原master重新加入redis集群
[root@redis-master ~]#vim /etc/redis.conf replicaof 10.0.0.18 6379 masterauth 123456 [root@redis-master ~]#systemctl start redis
| 1 2 3 4 5 6 | [ root @ redis - master ~ ] #vim /etc/redis.conf replicaof 10.0.0.18 6379 masterauth 123456 [ root @ redis - master ~ ] #systemctl start redis |
在原master上观察状态
[root@redis-master ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> INFO replication # Replication role:slave master_host:10.0.0.18 master_port:6379 master_link_status:up master_last_io_seconds_ago:0 master_sync_in_progress:0 slave_repl_offset:764754 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:75e3f205082c5a10824fbe6580b6ad4437140b94 master_replid2:b2fb4653bdf498691e5f88519ded65b6c000e25c master_repl_offset:764754 second_repl_offset:46451 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:46451 repl_backlog_histlen:718304 [root@redis-master ~]#redis-cli -a 123456 -p 26379 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:26379> INFO sentinel # Sentinel sentinel_masters:1 sentinel_tilt:0 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 sentinel_simulate_failure_flags:0 master0:name=mymaster,status=ok,address=10.0.0.18:6379,slaves=2,sentinels=2 127.0.0.1:26379>
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | [ root @ redis - master ~ ] #redis-cli -a 123456 Warning : Using a password with '-a' or '-u' option on the command line interface may not be safe . 127.0.0.1 : 6379 > INFO replication # Replication role : slave master_host : 10.0.0.18 master_port : 6379 master_link_status : up master_last_io_seconds_ago : 0 master_sync_in_progress : 0 slave_repl_offset : 764754 slave_priority : 100 slave_read_only : 1 connected_slaves : 0 master_replid : 75e3f205082c5a10824fbe6580b6ad4437140b94 master_replid2 : b2fb4653bdf498691e5f88519ded65b6c000e25c master_repl_offset : 764754 second_repl_offset : 46451 repl_backlog_active : 1 repl_backlog_size : 1048576 repl_backlog_first_byte_offset : 46451 repl_backlog_histlen : 718304 [ root @ redis - master ~ ] #redis-cli -a 123456 -p 26379 Warning : Using a password with '-a' or '-u' option on the command line interface may not be safe . 127.0.0.1 : 26379 > INFO sentinel # Sentinel sentinel_masters : 1 sentinel_tilt : 0 sentinel_running_scripts : 0 sentinel_scripts_queue_length : 0 sentinel_simulate_failure_flags : 0 master0 : name = mymaster , status = ok , address = 10.0.0.18 : 6379 , slaves = 2 , sentinels = 2 127.0.0.1 : 26379 > |
观察新master上状态和日志
[root@redis-slave1 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> INFO replication # Replication role:master connected_slaves:2 slave0:ip=10.0.0.28,port=6379,state=online,offset=769027,lag=0 slave1:ip=10.0.0.8,port=6379,state=online,offset=769027,lag=0 master_replid:75e3f205082c5a10824fbe6580b6ad4437140b94 master_replid2:b2fb4653bdf498691e5f88519ded65b6c000e25c master_repl_offset:769160 second_repl_offset:46451 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:287 repl_backlog_histlen:768874 127.0.0.1:6379> [root@redis-slave1 ~]#tail -f /var/log/redis/sentinel.log 25717:X 20 Feb 2020 17:42:33.757 # +sdown slave 10.0.0.8:6379 10.0.0.8 6379 @ mymaster 10.0.0.18 6379 25717:X 20 Feb 2020 18:41:29.566 # -sdown slave 10.0.0.8:6379 10.0.0.8 6379 @ mymaster 10.0.0.18 6379
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | [ root @ redis - slave1 ~ ] #redis-cli -a 123456 Warning : Using a password with '-a' or '-u' option on the command line interface may not be safe . 127.0.0.1 : 6379 > INFO replication # Replication role : master connected_slaves : 2 slave0 : ip = 10.0.0.28 , port = 6379 , state = online , offset = 769027 , lag = 0 slave1 : ip = 10.0.0.8 , port = 6379 , state = online , offset = 769027 , lag = 0 master_replid : 75e3f205082c5a10824fbe6580b6ad4437140b94 master_replid2 : b2fb4653bdf498691e5f88519ded65b6c000e25c master_repl_offset : 769160 second_repl_offset : 46451 repl_backlog_active : 1 repl_backlog_size : 1048576 repl_backlog_first_byte_offset : 287 repl_backlog_histlen : 768874 127.0.0.1 : 6379 > [ root @ redis - slave1 ~ ] #tail -f /var/log/redis/sentinel.log 25717 : X 20 Feb 2020 17 : 42 : 33.757 # +sdown slave 10.0.0.8:6379 10.0.0.8 6379 @ mymaster 10.0.0.18 6379 25717 : X 20 Feb 2020 18 : 41 : 29.566 # -sdown slave 10.0.0.8:6379 10.0.0.8 6379 @ mymaster 10.0.0.18 6379 |
应用程序如何连接redis
Redis 官方客户端：https://redis.io/clients
java 客户端连接Redis：https://github.com/xetorthio/jedis/blob/master/pom.xml
#jedis/pom.xml 配置连接redis <properties> <redis-hosts>localhost:6379,localhost:6380,localhost:6381,localhost:6382,localhost:6383,localhost:6384,localhost:6385</redis-hosts> <sentinel-hosts>localhost:26379,localhost:26380,localhost:26381</sentinel-hosts> <cluster-hosts>localhost:7379,localhost:7380,localhost:7381,localhost:7382,localhost:7383,localhost:7384,localhost:7385</cluster-hosts> <github.global.server>github</github.global.server> </properties>
| 1 2 3 4 5 6 7 | #jedis/pom.xml 配置连接redis < properties > < redis - hosts > localhost : 6379 , localhost : 6380 , localhost : 6381 , localhost : 6382 , localhost : 6383 , localhost : 6384 , localhost : 6385 < / redis - hosts > < sentinel - hosts > localhost : 26379 , localhost : 26380 , localhost : 26381 < / sentinel - hosts > < cluster - hosts > localhost : 7379 , localhost : 7380 , localhost : 7381 , localhost : 7382 , localhost : 7383 , localhost : 7384 , localhost : 7385 < / cluster - hosts > < github . global . server > github < / github . global . server > < / properties > |
java客户端连接redis是通过Jedis来实现的，java代码用的时候只要创建Jedis对象就可以建多个Jedis连接池来连接
redis，应用程序再直接调用连接池即可连接Redis。
而Redis为了保障高可用,服务一般都是Sentinel部署方式，当Redis服务中的主服务挂掉之后,会仲裁出另外一台
Slaves服务充当Master。这个时候,我们的应用即使使用了Jedis 连接池,Master服务挂了,我们的应用将还是无法连
接新的Master服务，为了解决这个问题, Jedis也提供了相应的Sentinel实现,能够在Redis Sentinel主从切换时候,通
知我们的应用,把我们的应用连接到新的Master服务。
Redis Sentinel的使用也是十分简单的,只是在JedisPool中添加了Sentinel和MasterName参数，JRedis Sentinel底
层基于Redis订阅实现Redis主从服务的切换通知，当Reids发生主从切换时，Sentinel会发送通知主动通知Jedis进
行连接的切换，JedisSentinelPool在每次从连接池中获取链接对象的时候,都要对连接对象进行检测,如果此链接和
Sentinel的Master服务连接参数不一致,则会关闭此连接,重新获取新的Jedis连接对象。
