Redis高可用与集群

redis 高可用与集群

虽然Redis可以实现单机的数据持久化，但无论是RDB也好或者AOF也好，都解决不了单点宕机问题，即一旦单台
redis服务器本身出现系统故障、硬件故障等问题后，就会直接造成数据的丢失，因此需要使用另外的技术来解决
单点问题。

配置 reids 主从复制

主备模式（master/slave），可以实现Redis数据的跨主机备份。
程序端连接到高可用负载的VIP，然后连接到负载服务器设置的Redis后端real server，此模式不需要在程序里面配 置Redis服务器的真实IP地址，当后期Redis服务器IP地址发生变更只需要更改redis 相应的后端real server即可， 可避免更改程序中的IP地址设置。

Slave主要配置

Redis Slave 也要开启持久化并设置和master同样的连接密码，因为后期slave会有提升为master的可能,Slave
端切换master同步后会丢失之前的所有数据。
一旦某个Slave成为一个master的slave，Redis Slave服务会清空当前redis服务器上的所有数据并将master的数据导入到自己的内存，但是断开同步关系后不会删除当前已经同步过的数据。

命令行配置

当前状态为master，需要转换为slave角色并指向master服务器的IP+PORT+Password

#在mater上设置key1 [root@centos8 ~]#redis-cli 127.0.0.1:6379> info NOAUTH Authentication required. 127.0.0.1:6379> AUTH 123456 OK 127.0.0.1:6379> INFO replication # Replication role:master connected_slaves:0 master_replid:a3504cab4d33e9723a7bc988ff8e022f6d9325bf master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0 127.0.0.1:6379> SET key1 v1-master OK 127.0.0.1:6379> KEYS * 1) "key1" 127.0.0.1:6379> GET key1 "v1-master" 127.0.0.1:6379> #以下都在slave上执行，登录 [root@centos8 ~]#redis-cli 127.0.0.1:6379> info NOAUTH Authentication required. 127.0.0.1:6379> AUTH 123456 OK 127.0.0.1:6379> INFO replication # Replication role:master connected_slaves:0 master_replid:a3504cab4d33e9723a7bc988ff8e022f6d9325bf master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0 127.0.0.1:6379> SET key1 v1-slave OK 127.0.0.1:6379> KEYS * 1) "key1" 127.0.0.1:6379> GET key1 "v1-slave" 127.0.0.1:6379> 127.0.0.1:6379> KEYS * 1) "key1" 127.0.0.1:6379> GET key1 "v1-18" 127.0.0.1:6379> 127.0.0.1:6379> INFO replication # Replication role:master connected_slaves:0 master_replid:a3504cab4d33e9723a7bc988ff8e022f6d9325bf master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0 127.0.0.1:6379> #在slave上设置master的IP和端口，4.0版之前的指令为slaveof 127.0.0.1:6379> REPLICAOF 10.0.0.8 6379 #仍可使用SLAVEOF MasterIP Port OK #在slave上设置master的密码，才可以同步 127.0.0.1:6379> CONFIG SET masterauth 123456 OK 127.0.0.1:6379> INFO replication # Replication #角色变为slave role:slave master_host:10.0.0.8 #指向master master_port:6379 master_link_status:up master_last_io_seconds_ago:8 master_sync_in_progress:0 slave_repl_offset:42 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:b69908f23236fb20b810d198f7f4539f795e0ee5 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:42 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:42 #查看已经同步成功 127.0.0.1:6379> GET key1 "v1-master"

同步日志

在master 上观察日志

[root@centos8 ~]#tail /var/log/redis/redis.log 11680:M 20 Feb 2020 11:33:21.349 * Background saving started by pid 11712 11712:C 20 Feb 2020 11:33:21.374 * DB saved on disk 11712:C 20 Feb 2020 11:33:21.374 * RDB: 4 MB of memory used by copy-on-write 11680:M 20 Feb 2020 11:33:21.475 * Background saving terminated with success 11680:M 20 Feb 2020 11:33:21.476 * Synchronization with replica 10.0.0.18:6379 succeeded 11680:M 20 Feb 2020 11:48:22.085 * 1 changes in 900 seconds. Saving... 11680:M 20 Feb 2020 11:48:22.101 * Background saving started by pid 11739 11739:C 20 Feb 2020 11:48:22.106 * DB saved on disk 11739:C 20 Feb 2020 11:48:22.107 * RDB: 2 MB of memory used by copy-on-write 11680:M 20 Feb 2020 11:48:22.202 * Background saving terminated with success

在slave节点观察日志

[root@centos8 ~]#tail -f /var/log/redis/redis.log

保存配置到redis.conf

[root@centos8 ~]#vim /etc/redis.conf ....... 285 # 286 # replicaof <masterip> <masterport> 287 replicaof 10.0.0.8 6379 #指定master的IP和端口号 288 # If the master is password protected (using the "requirepass" configuration 289 # directive below) it is possible to tell the replica to authenticate before 290 # starting the replication synchronization process, otherwise the master will 291 # refuse the replica request. 292 # 293 # masterauth <master-password> 294 masterauth 123456 #如果密码需要设置 ....... [root@centos8 ~]#systemctl restart redis

master和slave查看状态

#在master上查看状态 127.0.0.1:6379> info replication # Replication role:master connected_slaves:1 slave0:ip=10.0.0.18,port=6379,state=online,offset=1104403,lag=0 master_replid:b2517cd6cb3ad1508c516a38caed5b9d2d9a3e73 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:1104403 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:55828 repl_backlog_histlen:1048576 127.0.0.1:6379> #在slave上查看状态 127.0.0.1:6379> INFO replication # Replication role:slave master_host:10.0.0.8 master_port:6379 master_link_status:up master_last_io_seconds_ago:6 master_sync_in_progress:0 slave_repl_offset:1104431 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:b2517cd6cb3ad1508c516a38caed5b9d2d9a3e73 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:1104431 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:55856 repl_backlog_histlen:1048576 127.0.0.1:6379> #停止master的redis服务：systemctl stop redis,在slave上可以观察到以下现象 127.0.0.1:6379> INFO replication # Replication role:slave master_host:10.0.0.8 master_port:6379 master_link_status:down #显示down，表示无法连接master master_last_io_seconds_ago:-1 master_sync_in_progress:0 slave_repl_offset:1104529 master_link_down_since_seconds:4 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:b2517cd6cb3ad1508c516a38caed5b9d2d9a3e73 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:1104529 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:55954 repl_backlog_histlen:1048576 127.0.0.1:6379>

slave观察日志

[root@centos8 ~]#tail -f /var/log/redis/redis.log 24592:S 20 Feb 2020 12:03:58.792 * Connecting to MASTER 10.0.0.8:6379 24592:S 20 Feb 2020 12:03:58.792 * MASTER <-> REPLICA sync started 24592:S 20 Feb 2020 12:03:58.797 * Non blocking connect for SYNC fired the event. 24592:S 20 Feb 2020 12:03:58.797 * Master replied to PING, replication can continue... 24592:S 20 Feb 2020 12:03:58.798 * Partial resynchronization not possible (no cached master) 24592:S 20 Feb 2020 12:03:58.801 * Full resync from master: b69908f23236fb20b810d198f7f4539f795e0ee5:2440 24592:S 20 Feb 2020 12:03:58.863 * MASTER <-> REPLICA sync: receiving 213 bytes from master 24592:S 20 Feb 2020 12:03:58.863 * MASTER <-> REPLICA sync: Flushing old data 24592:S 20 Feb 2020 12:03:58.863 * MASTER <-> REPLICA sync: Loading DB in memory 24592:S 20 Feb 2020 12:03:58.863 * MASTER <-> REPLICA sync: Finished with success

slave 状态只读无法写入数据

127.0.0.1:6379> set key1 v1-slave (error) READONLY You can't write against a read only replica.

Master日志

[root@centos8 ~]#tail /var/log/redis/redis.log 11846:M 20 Feb 2020 12:11:35.171 * DB loaded from disk: 0.000 seconds 11846:M 20 Feb 2020 12:11:35.171 * Ready to accept connections 11846:M 20 Feb 2020 12:11:36.086 * Replica 10.0.0.18:6379 asks for synchronization 11846:M 20 Feb 2020 12:11:36.086 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'b69908f23236fb20b810d198f7f4539f795e0ee5', my replication IDs are '4bff970970c073c1f3d8e8ad20b1c1f126a5f31c' and '0000000000000000000000000000000000000000') 11846:M 20 Feb 2020 12:11:36.086 * Starting BGSAVE for SYNC with target: disk 11846:M 20 Feb 2020 12:11:36.095 * Background saving started by pid 11850 11850:C 20 Feb 2020 12:11:36.121 * DB saved on disk 11850:C 20 Feb 2020 12:11:36.121 * RDB: 4 MB of memory used by copy-on-write 11846:M 20 Feb 2020 12:11:36.180 * Background saving terminated with success 11846:M 20 Feb 2020 12:11:36.180 * Synchronization with replica 10.0.0.18:6379 succeeded

主从复制过程

Redis支持主从复制分为全量同步和增量同步，首次同步是全量同步，主从同步可以让从服务器从主服务器备份数
据，而且从服务器还可与有从服务器，即另外一台redis服务器可以从一台从服务器进行数据同步，redis 的主从同 步是非阻塞的，master收到从服务器的sync(2.8版本之前是PSYNC)命令会fork一个子进程在后台执行bgsave命
令，并将新写入的数据写入到一个缓冲区里面，bgsave执行完成之后并生成的将RDB文件发送给客户端，客户端将 收到后的RDB文件载入自己的内存，然后redis master再将缓冲区的内容在全部发送给redis slave，之后的同步
slave服务器会发送一个offset的位置(等同于MySQL的binlog的位置)给主服务器，主服务器检查后位置没有错误将 此位置之后的数据包括写在缓冲区的积压数据发送给redis从服务器，从服务器将主服务器发送的积压数据写入内存，这样一次完整的数据同步，再之后同步的时候从服务器只要发送当前的offset位置给主服务器，然后主服务 器根据相应的位置将之后的数据发送给从服务器保存到其内存即可。 Redis全量复制一般发生在Slave首次初始化阶段，这时Slave需要将Master上的所有数据都复制一份。

具体主从同 步骤如下：

1）从服务器连接主服务器，发送SYNC命令 2）主服务器接收到SYNC命令后，开始执行BGSAVE命令生成RDB快照文件并使用缓冲区记录此后执行的所有写命令 3）主服务器BGSAVE执行完后，向所有从服务器发送RDB快照文件，并在发送期间继续记录被执行的写命令 4）从服务器收到快照文件后丢弃所有旧数据，载入收到的快照 5）主服务器快照发送完毕后开始向从服务器发送缓冲区中的写命令 6）从服务器完成对快照的载入，开始接收命令请求，并执行来自主服务器缓冲区的写命令 7）后期同步会先发送自己slave_repl_offset位置，只同步新增加的数据，不再全量同步

主从同步优化配置

Redis在2.8版本之前没有提供增量部分复制的功能，当网络闪断或者slave Redis重启之后会导致主从之间的全量同 步，即从2.8版本开始增加了部分复制的功能。

性能相关配置

repl-diskless-sync no # 是否使用无盘同步RDB文件，默认为no，no为不使用无盘，需要将RDB文件保存到磁盘后 再发送给slave，yes为支持无盘，支持无盘就是RDB文件不需要保存至本地磁盘，而且直接通过socket文件发送给 slave repl-diskless-sync-delay 5 #diskless时复制的服务器等待的延迟时间 repl-ping-slave-period 10 #slave端向server端发送ping的时间区间设置，默认为10秒 repl-timeout 60 #设置超时时间 repl-disable-tcp-nodelay no #是否启用TCP_NODELAY，如设置成yes，则redis会合并小的TCP包从而节省带宽， 但会增加同步延迟（40ms），造成master与slave数据不一致，假如设置成no，则redis master会立即发送同步数据，没有延迟，前者关注性能，后者关注redis服务中的数据一致性 repl-backlog-size 1mb #master的写入数据缓冲区，用于记录自上一次同步后到下一次同步过程中间的写入命令，计算公式：repl-backlog-size = 允许从节点最大中断时长 * 主实例offset每秒写入量，比如master每秒最大写入64mb，最大允许60秒，那么就要设置为64mb*60秒=3840MB(3.8G) repl-backlog-ttl 3600 #如果一段时间后没有slave连接到master，则backlog size的内存将会被释放。如果值为0则 表示永远不释放这部份内存。 slave-priority 100 #slave端的优先级设置，值是一个整数，数字越小表示优先级越高。当master故障时将会按照优先级来选择slave端进行恢复，如果值设置为0，则表示该slave永远不会被选择。 min-replicas-to-write 1 #设置一个master的可用slave不能少于多少个，否则无法master写 min-slaves-max-lag 20 #设置至少有上面数量的slave延迟时间都大于多少秒时，master不接收写操作(拒绝写入)

Slave同步过程日志

master同步日志

当master故障后，可以手动将slave 提升为master

master故障后，只能手动提升一个slave为新master，不支持自动切换。Master的切换会导致master_replid发生变化，slave之前的master_replid就和当前master不一致从而会引发所有 slave的全量同步。

#查看当前状态为slave 127.0.0.1:6379> INFO replication # Replication role:slave master_host:10.0.0.18 master_port:6379 master_link_status:up master_last_io_seconds_ago:1 master_sync_in_progress:0 slave_repl_offset:3794 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:8e8279e461fdf0f1a3464ef768675149ad4b54a3 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:3794 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:3781 repl_backlog_histlen:14 127.0.0.1:6379>

停止slave同步并查看当前状态：

#提升为master角色 127.0.0.1:6379> REPLICAOF NO ONE #旧版使用SLAVEOF no one OK (5.04s) 127.0.0.1:6379> info replication # Replication role:master connected_slaves:0 master_replid:94901d6b8ff812ec4a4b3ac6bb33faa11e55c274 master_replid2:0083e5a9c96aa4f2196934e10b910937d82b4e19 master_repl_offset:3514 second_repl_offset:3515 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:3431 repl_backlog_histlen:84 127.0.0.1:6379>

测试能否写入数据：

127.0.0.1:6379> set keytest1 vtest1 OK

修改指向新的master

127.0.0.1:6379> SLAVEOF 10.0.0.8 6379 OK 127.0.0.1:6379> set key100 v100 (error) READONLY You can't write against a read only replica. #查看日志 [root@centos8 ~]#tail -f /var/log/redis/redis.log 1762:S 20 Feb 2020 13:28:21.943 # Connection with master lost. 1762:S 20 Feb 2020 13:28:21.943 * Caching the disconnected master state. 1762:S 20 Feb 2020 13:28:21.943 * REPLICAOF 10.0.0.8:6379 enabled (user request from 'id=5 addr=127.0.0.1:59668 fd=9 name= age=149 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=41 qbuf-free=32727 obl=0 oll=0 omem=0 events=r cmd=slaveof') 1762:S 20 Feb 2020 13:28:21.966 * Connecting to MASTER 10.0.0.8:6379 1762:S 20 Feb 2020 13:28:21.966 * MASTER <-> REPLICA sync started 1762:S 20 Feb 2020 13:28:21.967 * Non blocking connect for SYNC fired the event. 1762:S 20 Feb 2020 13:28:21.968 * Master replied to PING, replication can continue... 1762:S 20 Feb 2020 13:28:21.968 * Trying a partial resynchronization (request 8e8279e461fdf0f1a3464ef768675149ad4b54a3:3991). 1762:S 20 Feb 2020 13:28:21.969 * Successful partial resynchronization with master. 1762:S 20 Feb 2020 13:28:21.969 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

在master可看到有两个slave

repl_backlog_histlen:1048576 127.0.0.1:6379> INFO replication # Replication role:master connected_slaves:2 slave0:ip=10.0.0.18,port=6379,state=online,offset=4606,lag=0 #两个slave节点 slave1:ip=10.0.0.28,port=6379,state=online,offset=4606,lag=0 master_replid:8e8279e461fdf0f1a3464ef768675149ad4b54a3 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:4606 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:4606 127.0.0.1:6379>

Slave节点再有Slave

在有slave的”master”查看状态，也就是中间那个slave：

127.0.0.1:6379> INFO replication # Replication role:slave master_host:10.0.0.8 master_port:6379 master_link_status:up master_last_io_seconds_ago:8 #最近一次与master通信已经过去多少秒。 master_sync_in_progress:0 #是否正在与master通信。 slave_repl_offset:4312 #当前同步的偏移量 slave_priority:100 #slave优先级，master故障后值越小越优先同步。 slave_read_only:1 connected_slaves:1 slave0:ip=10.0.0.28,port=6379,state=online,offset=4312,lag=0 #slave的slave节点 master_replid:8e8279e461fdf0f1a3464ef768675149ad4b54a3 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:4312 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:4312

常见主从复制故障汇总

master密码不对

即配置的master密码不对，导致验证不通过而无法建立主从同步关系。

[root@centos8 ~]#tail -f /var/log/redis/redis.log 24930:S 20 Feb 2020 13:53:57.029 * Connecting to MASTER 10.0.0.8:6379 24930:S 20 Feb 2020 13:53:57.030 * MASTER <-> REPLICA sync started 24930:S 20 Feb 2020 13:53:57.030 * Non blocking connect for SYNC fired the event. 24930:S 20 Feb 2020 13:53:57.030 * Master replied to PING, replication can continue... 24930:S 20 Feb 2020 13:53:57.031 # Unable to AUTH to MASTER: -ERR invalid password

Redis版本不一致

不同的redis 大版本之间存在兼容性问题，比如：3和4，4和5之间，因此各master和slave之间必须保持版本一致

无法远程连接

在开启了安全模式情况下，没有设置bind地址或者密码。

[root@centos8 ~]#vim /etc/redis.conf #bind 127.0.0.1 #将此行注释 [root@centos8 ~]#systemctl restart redis [root@centos8 ~]#ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 100 127.0.0.1:25 0.0.0.0:* LISTEN 0 128 0.0.0.0:6379 0.0.0.0:* LISTEN 0 128 [::]:22 [::]:* LISTEN 0 100 [::1]:25 [::]:* LISTEN 0 128 [::]:6379 [::]:* [root@centos8 ~]# [root@centos8 ~]#redis-cli -h 10.0.0.8 10.0.0.8:6379> KEYS * (error) DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside. 10.0.0.38:6379> exit #可以本机登录 [root@centos8 ~]#redis-cli 127.0.0.1:6379> KEYS * (empty list or set)