Redis Cluster集群维护：动态扩容

Redis cluster集群节点维护

集群运行时间长久之后，难免由于硬件故障、网络规划、业务增长等原因对已有集群进行相应的调整， 比如增加
Redis node节点、减少节点、节点迁移、更换服务器等。
增加节点和删除节点会涉及到已有的槽位重新分配及数据迁移。

集群维护之动态扩容

实战案例：
因公司业务发展迅猛，现有的三主三从的redis cluster架构可能无法满足现有业务的并发写入需求，因此公司紧急采购两台服务器10.0.0.68，10.0.0.78，需要将其动态添加到集群当中，但不能影响业务使用和数据丢失。

添加节点准备

增加Redis node节点，需要与之前的Redis node版本相同、配置一致，然后分别再启动两台Redis node，应为一主一从。

#配置node7节点 [root@redis-node7 ~]#dnf -y install redis [root@redis-node7 ~]#sed -i.bak -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e '/masterauth/a masterauth 123456' -e '/# requirepass/a requirepass 123456' -e '/# cluster-enabled yes/a cluster-enabled yes' -e '/# cluster-config-file nodes-6379.conf/a cluster-config-file nodes-6379.conf' -e '/appendonly no/c appendonly yes' /etc/redis.conf [root@redis-node7 ~]#systemctl start redis #配置node8节点 [root@redis-node8 ~]#dnf -y install redis [root@redis-node8 ~]#sed -i.bak -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e '/masterauth/a masterauth 123456' -e '/# requirepass/a requirepass 123456' -e '/# cluster-enabled yes/a cluster-enabled yes' -e '/# cluster-config-file nodes-6379.conf/a cluster-config-file nodes-6379.conf' -e '/appendonly no/c appendonly yes' /etc/redis.conf [root@redis-node8 ~]#systemctl start redis

添加节点到集群

使用以下命令添加新节点，要添加的新redis节点IP和端口添加到的已有的集群中任意节点的IP:端口

add-node new_host:new_port existing_host:existing_port 说明： new_host:new_port #为新添加的主机的IP和端口 existing_host:existing_port #为已有的集群中任意节点的IP和端口

Redis 4添加方式：
把新的Redis 节点192.168.7.107添加到当前Redis集群当中。

[root@redis-node1 ~]# redis-trib.rb add-node 10.0.0.68:6379 10.0.0.58:6379

Redis 5添加方式：

#将第一台新的主机都加入集群 [root@redis-node1 ~]#redis-cli -a 123456 --cluster add-node 10.0.0.68:6379 10.0.0.58:6379 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. >>> Adding node 10.0.0.68:6379 to cluster 10.0.0.58:6379 >>> Performing Cluster Check (using node 10.0.0.58:6379) S: 9875b50925b4e4f29598e6072e5937f90df9fc71 10.0.0.58:6379 slots: (0 slots) slave replicates d34da8666a6f587283a1c2fca5d13691407f9462 M: d04e524daec4d8e22bdada7f21a9487c2d3e1057 10.0.0.48:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) M: d34da8666a6f587283a1c2fca5d13691407f9462 10.0.0.28:6379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) S: 99720241248ff0e4c6fa65c2385e92468b3b5993 10.0.0.18:6379 slots: (0 slots) slave replicates d04e524daec4d8e22bdada7f21a9487c2d3e1057 S: f9adcfb8f5a037b257af35fa548a26ffbadc852d 10.0.0.38:6379 slots: (0 slots) slave replicates cb028b83f9dc463d732f6e76ca6bbcd469d948a7 M: cb028b83f9dc463d732f6e76ca6bbcd469d948a7 10.0.0.8:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. >>> Send CLUSTER MEET to node 10.0.0.68:6379 to make it join the cluster. [OK] New node added correctly. #观察到已经加入成功，但无没有slot位，而且新的主机是master [root@redis-node1 ~]#redis-cli -a 123456 --cluster info 10.0.0.8:6379 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 10.0.0.8:6379 (cb028b83...) -> 6672 keys | 5461 slots | 1 slaves. 10.0.0.78:6379 (f67f1c02...) -> 0 keys | 0 slots | 0 slaves. 10.0.0.68:6379 (d6e2eca6...) -> 0 keys | 0 slots | 0 slaves. 10.0.0.48:6379 (d04e524d...) -> 6679 keys | 5462 slots | 1 slaves. 10.0.0.28:6379 (d34da866...) -> 6649 keys | 5461 slots | 1 slaves. [OK] 20000 keys in 5 masters. 1.22 keys per slot on average. [root@redis-node1 ~]#redis-cli -a 123456 --cluster check 10.0.0.8:6379 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 10.0.0.8:6379 (cb028b83...) -> 6672 keys | 5461 slots | 1 slaves. 10.0.0.78:6379 (f67f1c02...) -> 0 keys | 0 slots | 0 slaves. 10.0.0.68:6379 (d6e2eca6...) -> 0 keys | 0 slots | 0 slaves. 10.0.0.48:6379 (d04e524d...) -> 6679 keys | 5462 slots | 1 slaves. 10.0.0.28:6379 (d34da866...) -> 6649 keys | 5461 slots | 1 slaves. [OK] 20000 keys in 5 masters. 1.22 keys per slot on average. >>> Performing Cluster Check (using node 10.0.0.8:6379) M: cb028b83f9dc463d732f6e76ca6bbcd469d948a7 10.0.0.8:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) M: d6e2eca6b338b717923f64866bd31d42e52edc98 10.0.0.68:6379 slots: (0 slots) master S: 9875b50925b4e4f29598e6072e5937f90df9fc71 10.0.0.58:6379 slots: (0 slots) slave replicates d34da8666a6f587283a1c2fca5d13691407f9462 S: f9adcfb8f5a037b257af35fa548a26ffbadc852d 10.0.0.38:6379 slots: (0 slots) slave replicates cb028b83f9dc463d732f6e76ca6bbcd469d948a7 M: d04e524daec4d8e22bdada7f21a9487c2d3e1057 10.0.0.48:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 99720241248ff0e4c6fa65c2385e92468b3b5993 10.0.0.18:6379 slots: (0 slots) slave replicates d04e524daec4d8e22bdada7f21a9487c2d3e1057 M: d34da8666a6f587283a1c2fca5d13691407f9462 10.0.0.28:6379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@redis-node1 ~]#cat /var/lib/redis/nodes-6379.conf d6e2eca6b338b717923f64866bd31d42e52edc98 10.0.0.68:6379@16379 master - 0 1582356107260 8 connected 9875b50925b4e4f29598e6072e5937f90df9fc71 10.0.0.58:6379@16379 slave d34da8666a6f587283a1c2fca5d13691407f9462 0 1582356110286 6 connected f9adcfb8f5a037b257af35fa548a26ffbadc852d 10.0.0.38:6379@16379 slave cb028b83f9dc463d732f6e76ca6bbcd469d948a7 0 1582356108268 4 connected d04e524daec4d8e22bdada7f21a9487c2d3e1057 10.0.0.48:6379@16379 master - 0 1582356105000 7 connected 5461-10922 99720241248ff0e4c6fa65c2385e92468b3b5993 10.0.0.18:6379@16379 slave d04e524daec4d8e22bdada7f21a9487c2d3e1057 0 1582356108000 7 connected d34da8666a6f587283a1c2fca5d13691407f9462 10.0.0.28:6379@16379 master - 0 1582356107000 3 connected 10923-16383 cb028b83f9dc463d732f6e76ca6bbcd469d948a7 10.0.0.8:6379@16379 myself,master - 0 1582356106000 1 connected 0-5460 vars currentEpoch 8 lastVoteEpoch 7 #和上面显示结果一样 [root@redis-node1 ~]#redis-cli -a 123456 CLUSTER NODES Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. d6e2eca6b338b717923f64866bd31d42e52edc98 10.0.0.68:6379@16379 master - 0 1582356313200 8 connected 9875b50925b4e4f29598e6072e5937f90df9fc71 10.0.0.58:6379@16379 slave d34da8666a6f587283a1c2fca5d13691407f9462 0 1582356311000 6 connected f9adcfb8f5a037b257af35fa548a26ffbadc852d 10.0.0.38:6379@16379 slave cb028b83f9dc463d732f6e76ca6bbcd469d948a7 0 1582356314208 4 connected d04e524daec4d8e22bdada7f21a9487c2d3e1057 10.0.0.48:6379@16379 master - 0 1582356311182 7 connected 5461-10922 99720241248ff0e4c6fa65c2385e92468b3b5993 10.0.0.18:6379@16379 slave d04e524daec4d8e22bdada7f21a9487c2d3e1057 0 1582356312000 7 connected d34da8666a6f587283a1c2fca5d13691407f9462 10.0.0.28:6379@16379 master - 0 1582356312190 3 connected 10923-16383 cb028b83f9dc463d732f6e76ca6bbcd469d948a7 10.0.0.8:6379@16379 myself,master - 0 1582356310000 1 connected 0-5460 #查看集群状态 [root@redis-node1 ~]#redis-cli -a 123456 CLUSTER INFO Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:7 cluster_size:3 cluster_current_epoch:8 cluster_my_epoch:1 cluster_stats_messages_ping_sent:17442 cluster_stats_messages_pong_sent:13318 cluster_stats_messages_fail_sent:4 cluster_stats_messages_auth-ack_sent:1 cluster_stats_messages_sent:30765 cluster_stats_messages_ping_received:13311 cluster_stats_messages_pong_received:13367 cluster_stats_messages_meet_received:7 cluster_stats_messages_fail_received:1 cluster_stats_messages_auth-req_received:1 cluster_stats_messages_received:26687 [root@redis-node1 ~]#

重新分配槽位

新的node节点加到集群之后默认是master节点，但是没有slots数据，需要重新分配

添加主机之后需要对添加至集群种的新主机重新分片否则其没有分片也就无法写入数据。
验证当前状态：

redis4的命令

[root@redis-node1 ~]# redis-trib.rb check 10.0.0.68:6379 #当前状态 [root@redis-node1 ~]# redis-trib.rb reshard 10.0.0.68:6379 #重新分片

Redis 5：

[root@redis-node1 ~]#redis-cli -a 123456 --cluster reshard 10.0.0.68:6379 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. >>> Performing Cluster Check (using node 10.0.0.68:6379) M: d6e2eca6b338b717923f64866bd31d42e52edc98 10.0.0.68:6379 slots: (0 slots) master M: d34da8666a6f587283a1c2fca5d13691407f9462 10.0.0.28:6379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) M: d04e524daec4d8e22bdada7f21a9487c2d3e1057 10.0.0.48:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) M: cb028b83f9dc463d732f6e76ca6bbcd469d948a7 10.0.0.8:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) S: 99720241248ff0e4c6fa65c2385e92468b3b5993 10.0.0.18:6379 slots: (0 slots) slave replicates d04e524daec4d8e22bdada7f21a9487c2d3e1057 M: f67f1c02c742cd48d3f48d8c362f9f1b9aa31549 10.0.0.78:6379 slots: (0 slots) master S: f9adcfb8f5a037b257af35fa548a26ffbadc852d 10.0.0.38:6379 slots: (0 slots) slave replicates cb028b83f9dc463d732f6e76ca6bbcd469d948a7 S: 9875b50925b4e4f29598e6072e5937f90df9fc71 10.0.0.58:6379 slots: (0 slots) slave replicates d34da8666a6f587283a1c2fca5d13691407f9462 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)?4096 #分配多少个槽位=16384/master个数 What is the receiving node ID? d6e2eca6b338b717923f64866bd31d42e52edc98 #10.0.0.68的ID Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1: all #将哪些源主机的槽位分配给新的节点，all是自动在所有的redis node选择划分，如果是从redis cluster删除某个主机可以使用此方式将指定主机上的槽位全部移动到别的redis主机 ...... Do you want to proceed with the proposed reshard plan (yes/no)? yes #确认分配 ...... Moving slot 12280 from 10.0.0.28:6379 to 10.0.0.68:6379: . Moving slot 12281 from 10.0.0.28:6379 to 10.0.0.68:6379: . Moving slot 12282 from 10.0.0.28:6379 to 10.0.0.68:6379: Moving slot 12283 from 10.0.0.28:6379 to 10.0.0.68:6379: .. Moving slot 12284 from 10.0.0.28:6379 to 10.0.0.68:6379: Moving slot 12285 from 10.0.0.28:6379 to 10.0.0.68:6379: . Moving slot 12286 from 10.0.0.28:6379 to 10.0.0.68:6379: Moving slot 12287 from 10.0.0.28:6379 to 10.0.0.68:6379: .. [root@redis-node1 ~]# #确定slot分配成功 [root@redis-node1 ~]#redis-cli -a 123456 --cluster check 10.0.0.8:6379 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 10.0.0.8:6379 (cb028b83...) -> 5019 keys | 4096 slots | 1 slaves. 10.0.0.68:6379 (d6e2eca6...) -> 4948 keys | 4096 slots | 0 slaves. 10.0.0.48:6379 (d04e524d...) -> 5033 keys | 4096 slots | 1 slaves. 10.0.0.28:6379 (d34da866...) -> 5000 keys | 4096 slots | 1 slaves. [OK] 20000 keys in 5 masters. 1.22 keys per slot on average. >>> Performing Cluster Check (using node 10.0.0.8:6379) M: cb028b83f9dc463d732f6e76ca6bbcd469d948a7 10.0.0.8:6379 slots:[1365-5460] (4096 slots) master 1 additional replica(s) M: d6e2eca6b338b717923f64866bd31d42e52edc98 10.0.0.68:6379 slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master #可看到4096个slots S: 9875b50925b4e4f29598e6072e5937f90df9fc71 10.0.0.58:6379 slots: (0 slots) slave replicates d34da8666a6f587283a1c2fca5d13691407f9462 S: f9adcfb8f5a037b257af35fa548a26ffbadc852d 10.0.0.38:6379 slots: (0 slots) slave replicates cb028b83f9dc463d732f6e76ca6bbcd469d948a7 M: d04e524daec4d8e22bdada7f21a9487c2d3e1057 10.0.0.48:6379 slots:[6827-10922] (4096 slots) master 1 additional replica(s) S: 99720241248ff0e4c6fa65c2385e92468b3b5993 10.0.0.18:6379 slots: (0 slots) slave replicates d04e524daec4d8e22bdada7f21a9487c2d3e1057 M: d34da8666a6f587283a1c2fca5d13691407f9462 10.0.0.28:6379 slots:[12288-16383] (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.

为新的master添加slave节点

需要再向当前的Redis集群中添加一个Redis单机服务器10.0.0.78，用于解决当前10.0.0.68单机的潜在宕机问题，即实现响应的高可用功能，有两种方式：

方法1：在新加节点到集群时，直接将之设置为slave

查看当前状态 [root@redis-node1 ~]#redis-cli -a 123456 --cluster check 10.0.0.8:6379 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 10.0.0.8:6379 (cb028b83...) -> 5019 keys | 4096 slots | 1 slaves. 10.0.0.68:6379 (d6e2eca6...) -> 4948 keys | 4096 slots | 0 slaves. 10.0.0.48:6379 (d04e524d...) -> 5033 keys | 4096 slots | 1 slaves. 10.0.0.28:6379 (d34da866...) -> 5000 keys | 4096 slots | 1 slaves. [OK] 20000 keys in 4 masters. 1.22 keys per slot on average. >>> Performing Cluster Check (using node 10.0.0.8:6379) M: cb028b83f9dc463d732f6e76ca6bbcd469d948a7 10.0.0.8:6379 slots:[1365-5460] (4096 slots) master 1 additional replica(s) M: d6e2eca6b338b717923f64866bd31d42e52edc98 10.0.0.68:6379 slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master S: 9875b50925b4e4f29598e6072e5937f90df9fc71 10.0.0.58:6379 slots: (0 slots) slave replicates d34da8666a6f587283a1c2fca5d13691407f9462 S: f9adcfb8f5a037b257af35fa548a26ffbadc852d 10.0.0.38:6379 slots: (0 slots) slave replicates cb028b83f9dc463d732f6e76ca6bbcd469d948a7 M: d04e524daec4d8e22bdada7f21a9487c2d3e1057 10.0.0.48:6379 slots:[6827-10922] (4096 slots) master 1 additional replica(s) S: 99720241248ff0e4c6fa65c2385e92468b3b5993 10.0.0.18:6379 slots: (0 slots) slave replicates d04e524daec4d8e22bdada7f21a9487c2d3e1057 M: d34da8666a6f587283a1c2fca5d13691407f9462 10.0.0.28:6379 slots:[12288-16383] (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. #直接加为slave节点 [root@redis-node1 ~]#redis-cli -a 123456 --cluster add-node 10.0.0.78:6379 10.0.0.8:6379 --cluster-slave --cluster-master-id d6e2eca6b338b717923f64866bd31d42e52edc98 #验证是否成功 [root@redis-node1 ~]#redis-cli -a 123456 --cluster check 10.0.0.8:6379 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 10.0.0.8:6379 (cb028b83...) -> 5019 keys | 4096 slots | 1 slaves. 10.0.0.68:6379 (d6e2eca6...) -> 4948 keys | 4096 slots | 1 slaves. 10.0.0.48:6379 (d04e524d...) -> 5033 keys | 4096 slots | 1 slaves. 10.0.0.28:6379 (d34da866...) -> 5000 keys | 4096 slots | 1 slaves. [OK] 20000 keys in 4 masters. 1.22 keys per slot on average. >>> Performing Cluster Check (using node 10.0.0.8:6379) M: cb028b83f9dc463d732f6e76ca6bbcd469d948a7 10.0.0.8:6379 slots:[1365-5460] (4096 slots) master 1 additional replica(s) M: d6e2eca6b338b717923f64866bd31d42e52edc98 10.0.0.68:6379 slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master 1 additional replica(s) S: 36840d7eea5835ba540d9b64ec018aa3f8de6747 10.0.0.78:6379 slots: (0 slots) slave replicates d6e2eca6b338b717923f64866bd31d42e52edc98 S: 9875b50925b4e4f29598e6072e5937f90df9fc71 10.0.0.58:6379 slots: (0 slots) slave replicates d34da8666a6f587283a1c2fca5d13691407f9462 S: f9adcfb8f5a037b257af35fa548a26ffbadc852d 10.0.0.38:6379 slots: (0 slots) slave replicates cb028b83f9dc463d732f6e76ca6bbcd469d948a7 M: d04e524daec4d8e22bdada7f21a9487c2d3e1057 10.0.0.48:6379 slots:[6827-10922] (4096 slots) master 1 additional replica(s) S: 99720241248ff0e4c6fa65c2385e92468b3b5993 10.0.0.18:6379 slots: (0 slots) slave replicates d04e524daec4d8e22bdada7f21a9487c2d3e1057 M: d34da8666a6f587283a1c2fca5d13691407f9462 10.0.0.28:6379 slots:[12288-16383] (4096 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.

方法2：先将新节点加入集群，再修改为slave

为新的master添加slave节点
命令格式：
Redis 3/4：

[root@redis-node1 ~]#redis-trib.rb add-node 10.0.0.78:6379 10.0.0.8:6379

Redis 5版本：
把10.0.0.78:6379添加到集群中：

[root@redis-node1 ~]#redis-cli -a 123456 --cluster add-node 10.0.0.78:6379 10.0.0.8:6379

更改新节点更改状态为slave：
需要手动将其指定为某个master的slave，否则其默认角色为master。

[root@redis-node1 ~]#redis-cli -h 10.0.0.78 -p 6339 -a 123456 #登录到新添加节点 10.0.0.78:6380> CLUSTER NODES #查看当前集群节点，找到目标master 的ID 10.0.0.78:6380> CLUSTER REPLICATE 886338acd50c3015be68a760502b239f4509881c #将其设置slave，命令格式为cluster replicate MASTERID 10.0.0.78:6380> CLUSTER NODES #再次查看集群节点状态，验证节点是否已经更改为指定master 的slave