Docker部署redis集群

igxiaoshan Lv5
  2023-10-30 09:10:29 2023-10-30 09:10:29 创建   2023-10-30 11:14:21 2023-10-30 11:14:21 更新      5.8k 字  28 分钟  

分布式存储案例

cluster(集群)模式-docker版

哈希槽分区进行亿级数据存储

经典问题

1~2亿条数据需要缓存,请问如何设计这个存储案例

单机单台100%不可能,必须用分布式存储

搭建redis分布式存储架构

解决方案

  • 哈希取余分区

2亿条记录就是2亿个k,v键值对;单机不行,必须要分布式多机存储

假设有3台机器构成一个集群,用户每次读写操作都是根据公式: hash(key) % N个机器台数,计算出哈希值,用来决定数据映射到哪一个节点上

优点

简单粗暴，直接有效，只需要预估好数据规划好节点，例如3台、8台、10台，就能保证一段时间的数据支撑。使用Hash算法让固定的一部分请求落到同一台服务器上，这样每台服务器固定处理一部分请求（并维护这些请求的信息），起到负载均衡 + 分而治之的作用

缺点

原来规划好的节点，进行扩容或者缩容就比较麻烦了，不管扩缩，每次数据变动导致节点有变动，映射关系需要重新进行计算，在服务器个数固定不变时没有问题，如果需要弹性扩容或者故障停机的情况下，原来的取模公式就会发生变化：Hash(key) /3 会变成Hash（key）/ ？。此时地址经过取余遇到某个redis机器宕机了，由于台数数量变化，会导致hash取余全部数据重新洗牌。

  • 一致性哈希算法分区

一致性哈希算法在1997年由麻省理工学院中提出的，设计目标是为了解决分布式缓存数据变动和映射问题，某个机器宕机了，分母数量改变了，自然取余数不OK了

作用

提出一致性Hash解决方案，目的是当服务器个数发生变动时，尽量减少影响客户端到服务器的映射关系

流程

步骤一: 算法构建一致性哈希环

一致性哈希环

一致性哈希算法必然有个hash函数并按照算法产生hash值，这个算法的所有可能哈希值会构成一个全量集，这个集合可以成为一个hash空间[0,2^32-1]，这是一个线性空间，但是在算法中，我们通过适当的逻辑控制将它首尾相连(0=2^32)，这样让它逻辑上形成一个环形空间

它也是按照使用取模的方法,前面笔记介绍的节点取模算法是对节点(服务器)的数量进行取模。而一致性Hash算法是对2^32取模，简单来说，一致性Hash算法将整个哈希值空间组织成一个虚拟的圆环，如假设某哈希函数H的值空间为0-2^32-1(即哈希值是一个32位无符号整形)，整个哈希环如上图：整个空间按顺时针方向组织，圆环的正上方的点代表0，0点右侧的第一个点代表1，以此类推2，3，4，…..直到2^32-1，也就是说0点左侧的第一个点代表2^32-1，0和2^32-1在零点中方向重合，我们把这个由2^32个点组成的圆环成为Hash环

步骤二: 服务器IP节点映射

节点映射

将集群中各个IP节点映射到环上的某一个位置

将各个服务器使用Hash进行一个哈希,集体可以选择服务器的ip或者主机名作为关键字进行哈希,这样每台机器就能确定其在哈希环上的位置。假如4个节点NodeA、B、C、D，经过IP地址的哈希函数计算（hash（ip）），使用IP地址哈希后在环空间的位置如下所示

步骤三: key落到服务器的落键规则

当我们需要存储一个K,V键值对时，首先计算key的hash值，hash（key），将这个key使用相同的函数hash计算出哈希值并确定此数据在环上的位置，从此位置沿着环顺时针‘行走’，第一台遇到的服务器就是其应该定位到的服务器，并将该键值对存储在该点上。

如，我们有Object A，Object B，Object C，Object D四个数据对象，经过哈希计算后，在环空间上的位置如下所示，根据一致性Hash算法，数据A会被定位到Node A上，B被定位到Node B上，C被定位到Node C上，D被定位到Node D上。

优点

优点一:

一致性哈希算法的容错性

容错性

假设Node C宕机，可以看到此时对象A，B，D不会受到影响，只有C对象被重定位到Node D。一般的，在一致性Hash算法中，如果一台服务器不可用，则受影响的数据仅仅是此服务器到环空间中前一台服务器（即沿着逆时针方向行走遇到的第一台服务）之间数据，其它不会受到影响。简单说，就是C挂了，收到影响的只是B、C之前的数据，并且这些数据会转移到D进行存储。如下图

优点二:

一致性哈希算法的扩展性

扩展性

服务器数量增加了，只需要加一台节点Node X，X的位置在A和B之间，那受到影响的也就是A和B之间的数据，重新把A和X的数据录入到X上即可，不会导致Hash取余全部数据重新洗牌。如下图

缺点

一致性哈希算法的数据倾斜问题

Hash环的数据倾斜问题

一致性哈希算法在服务节点太少时，容易因为节点分布不均匀而造成数据倾斜（被缓存的对象大部分集中缓存在某一台服务器上）问题。

如：系统中只有两台服务器，如下图所示

一致性哈希算法-小总结

为了在节点数目发生改变时尽可能少的迁移数据

将所有的存储节点排列在首尾相接的Hash环上，每个key在计算Hash后会顺时针找到临近的存储节点存放，而当有节点加入或退出时仅影响该节点在Hash环上顺时针相邻的后续节点。

优点

加入和删除节点只影响哈希环中顺时针方向相邻的节点，对其他节点无影响

缺点
数据的分布和节点的位置有关，因为这些节点不是均匀的分布在哈希环上的，所以数据在进行存储时达不到均匀分布的效果。

  • 哈希槽分区

哈希槽实质就是一个数组，数组 [0,2^14-1] 形成hash slot 空间

为什么会出现

解决一致性哈希算法的数据倾斜问题

作用

解决均匀分配的问题，在数据和节点之间又加入了一层，把这层称为哈希槽（slot），用于管理数据和节点之间的关系，现在就相当于节点上放的是槽，槽里放的是数据

槽解决的是粒度问题，相当于把粒度变大了，这样便于数据移动

哈希解决的是映射问题，使用key的哈希值来计算所在的槽，便于数据分配

多少个Hash槽

一个集群只能有16384个槽，编号0-16383（0-2^14-1)。这些槽会分配给集群中所在主节点，分配策略没有要求。可以指定哪些编号的槽分配给哪个主节点。集群会记录节点和槽对应关系。解决了节点和槽的关系后，接下来就需要对key求哈希值，然后对16384取余，余数是几，key就落入对应的槽里。slot=CRC16(key)%16384。以槽为单位移动数据，因为槽的数目是固定的，处理起来比较容易，这样数据移动问题就解决了。

哈希槽计算

Redis集群中内置了16384个哈希槽，redis会根据节点数量大致均等的将哈希槽映射到不同的节点。当需要在redis集群中放置一个key-value时，redis先对key使用CRC16算法算出一个结果，然后把结果对16384求余数，这样每个key都会对应一个编号在0-16383之间的哈希槽，也就是映射到某个节点上。

如下所示：key值A，B在Node2，key值C落在Node3上


Redis三主三从集群配置

前置条件

安装docker

docker启动—>systemctl start docker

创建redis实例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

docker run -d --name redis-node1 --net host --privileged=true -v /mydata/redis/cluster/redis-node-1:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6381

docker run -d --name redis-node2 --net host --privileged=true -v /mydata/redis/cluster/redis-node-2:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6382

docker run -d --name redis-node3 --net host --privileged=true -v /mydata/redis/cluster/redis-node-3:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6383

docker run -d --name redis-node4 --net host --privileged=true -v /mydata/redis/cluster/redis-node-4:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6384

docker run -d --name redis-node5 --net host --privileged=true -v /mydata/redis/cluster/redis-node-5:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6385

docker run -d --name redis-node6 --net host --privileged=true -v /mydata/redis/cluster/redis-node-6:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6386

### 命令分解
docker run  # 创建并运行docker容器实例
--name redis-node  # 容器名称
--net host  # 使用宿主机的IP和端口,默认
--privileged=true  # 获取宿主机root用户权限
-v /mydata/redis/share/redis-node:/data  # 容器卷,宿主机地址:docker内部地址
redis:6.0.8  # reids镜像和版本号
--cluster-enabled yes  # 开启redis集群
--appendonly yes  # 开启持久化
--port 6389  # redis 端口号

构建主从关系

redis-cli –cluster create 192.168.211.122:6381 192.168.211.122:6382 192.168.211.122:6383 192.168.211.122:6384 192.168.211.122:6385 192.168.211.122:6386 –cluster-replicas 1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
## 进入node1容器内部,构建主从关系
docker exec -it docker-node1 /bin/bash

redis-cli --cluster create 192.168.211.122:6381 192.168.211.122:6382 192.168.211.122:6383 192.168.211.122:6384 192.168.211.122:6385 192.168.211.122:6386 --cluster-replicas 1



### 示例
[root@CentOS7_001 ~]# docker exec -it 7695b8a6290c /bin/bash
root@CentOS7_001:/data# redis-cli --cluster create 192.168.211.122:6381 192.168.211.122:6382 192.168.211.122:6383 192.168.211.122:6384 192.168.211.122:6385 192.168.211.122:6386 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.211.122:6385 to 192.168.211.122:6381
Adding replica 192.168.211.122:6386 to 192.168.211.122:6382
Adding replica 192.168.211.122:6384 to 192.168.211.122:6383
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 36877ecd044077c08ff640c14eb60698fada9e5a 192.168.211.122:6381
   slots:[0-5460] (5461 slots) master
M: 6af5921ced502eae0ae93c991e1cafcee595e237 192.168.211.122:6382
   slots:[5461-10922] (5462 slots) master
M: 34fee68e4f9828340aea9dc71a2c2091981002ee 192.168.211.122:6383
   slots:[10923-16383] (5461 slots) master
S: 07bd7bc35e2a88624855481ee24ff89d8cfcf759 192.168.211.122:6384
   replicates 34fee68e4f9828340aea9dc71a2c2091981002ee
S: ac38a3672a052a72855157d03e5c1bf9b73c717e 192.168.211.122:6385
   replicates 36877ecd044077c08ff640c14eb60698fada9e5a
S: 16567fbe877b25e90f97d9b2af76ef80ca3427d8 192.168.211.122:6386
   replicates 6af5921ced502eae0ae93c991e1cafcee595e237
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 192.168.211.122:6381)
M: 36877ecd044077c08ff640c14eb60698fada9e5a 192.168.211.122:6381
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: ac38a3672a052a72855157d03e5c1bf9b73c717e 192.168.211.122:6385
   slots: (0 slots) slave
   replicates 36877ecd044077c08ff640c14eb60698fada9e5a
S: 16567fbe877b25e90f97d9b2af76ef80ca3427d8 192.168.211.122:6386
   slots: (0 slots) slave
   replicates 6af5921ced502eae0ae93c991e1cafcee595e237
S: 07bd7bc35e2a88624855481ee24ff89d8cfcf759 192.168.211.122:6384
   slots: (0 slots) slave
   replicates 34fee68e4f9828340aea9dc71a2c2091981002ee
M: 6af5921ced502eae0ae93c991e1cafcee595e237 192.168.211.122:6382
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 34fee68e4f9828340aea9dc71a2c2091981002ee 192.168.211.122:6383
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

  • 成功示例图,如下

查看集群状态

查看集群状态 cluster info

查看集群节点 cluster nodes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
## 连接进入6381客户端
redis-cli -p 6381

cluster info ## 查看集群状态
cluster nodes ## 查看集群节点

root@CentOS7_001:/data# redis-cli -p 6381
### 查看集群状态
127.0.0.1:6381> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:108
cluster_stats_messages_pong_sent:108
cluster_stats_messages_sent:216
cluster_stats_messages_ping_received:103
cluster_stats_messages_pong_received:108
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:216

### 查看集群节点信息
127.0.0.1:6381> cluster nodes
ac38a3672a052a72855157d03e5c1bf9b73c717e 192.168.211.122:6385@16385 slave 36877ecd044077c08ff640c14eb60698fada9e5a 0 1686884480383 1 connected
16567fbe877b25e90f97d9b2af76ef80ca3427d8 192.168.211.122:6386@16386 slave 6af5921ced502eae0ae93c991e1cafcee595e237 0 1686884479000 2 connected
07bd7bc35e2a88624855481ee24ff89d8cfcf759 192.168.211.122:6384@16384 slave 34fee68e4f9828340aea9dc71a2c2091981002ee 0 1686884481390 3 connected
6af5921ced502eae0ae93c991e1cafcee595e237 192.168.211.122:6382@16382 master - 0 1686884482000 2 connected 5461-10922
36877ecd044077c08ff640c14eb60698fada9e5a 192.168.211.122:6381@16381 myself,master - 0 1686884477000 1 connected 0-5460
34fee68e4f9828340aea9dc71a2c2091981002ee 192.168.211.122:6383@16383 master - 0 1686884482397 3 connected 10923-16383

读写存储

由于目前已经是集群模式

单机版连接客户端的方式redis-cli -p prot已经不适用了

需要使用集群版连接方式;,防止路由失效redis-cli -p port -c

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
## 单机连接
root@CentOS7_001:/data# redis-cli -p 6381
127.0.0.1:6381> keys *
(empty array)
127.0.0.1:6381> set k1 v1
(error) MOVED 12706 192.168.211.122:6383
127.0.0.1:6381> set k2 v2
OK
127.0.0.1:6381> set k3 v3
OK
127.0.0.1:6381> set k4 v4
(error) MOVED 8455 192.168.211.122:6382
127.0.0.1:6381> exit

### 无法正确路由,添加key失效


## 集群模式连接
root@CentOS7_001:/data# redis-cli -p 6381 -c
127.0.0.1:6381> keys *
1) "k3"
2) "k2"

#### 刷新掉所有的key
127.0.0.1:6381> flushall
OK
127.0.0.1:6381> keys *
(empty array)
127.0.0.1:6381> set k1 v1
-> Redirected to slot [12706] located at 192.168.211.122:6383
OK
192.168.211.122:6383> set k2 v2
-> Redirected to slot [449] located at 192.168.211.122:6381
OK
### 可以正常切换路由,添加key

查看集群信息

redis-cli --cluster check 192.168.211.112:6381

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
root@CentOS7_001:/data# redis-cli --cluster check 192.168.211.122:6381
192.168.211.122:6381 (36877ecd...) -> 0 keys | 5461 slots | 1 slaves.
192.168.211.122:6382 (6af5921c...) -> 0 keys | 5462 slots | 1 slaves.
192.168.211.122:6383 (34fee68e...) -> 1 keys | 5461 slots | 1 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.211.122:6381)
M: 36877ecd044077c08ff640c14eb60698fada9e5a 192.168.211.122:6381
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: ac38a3672a052a72855157d03e5c1bf9b73c717e 192.168.211.122:6385
   slots: (0 slots) slave
   replicates 36877ecd044077c08ff640c14eb60698fada9e5a
S: 16567fbe877b25e90f97d9b2af76ef80ca3427d8 192.168.211.122:6386
   slots: (0 slots) slave
   replicates 6af5921ced502eae0ae93c991e1cafcee595e237
S: 07bd7bc35e2a88624855481ee24ff89d8cfcf759 192.168.211.122:6384
   slots: (0 slots) slave
   replicates 34fee68e4f9828340aea9dc71a2c2091981002ee
M: 6af5921ced502eae0ae93c991e1cafcee595e237 192.168.211.122:6382
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 34fee68e4f9828340aea9dc71a2c2091981002ee 192.168.211.122:6383
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

容错切换迁移

停掉node1(master)节点,让node1下的slave上位

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
[root@CentOS7_001 ~]# docker ps
CONTAINER ID   IMAGE         COMMAND                   CREATED        STATUS       PORTS     NAMES
b69b4d4562ab   redis:6.0.8   "docker-entrypoint.s…"   47 hours ago   Up 4 hours             redis-node6
51cf2c0eb2bf   redis:6.0.8   "docker-entrypoint.s…"   47 hours ago   Up 4 hours             redis-node5
c00847c63342   redis:6.0.8   "docker-entrypoint.s…"   47 hours ago   Up 4 hours             redis-node4
4c9337895da0   redis:6.0.8   "docker-entrypoint.s…"   47 hours ago   Up 4 hours             redis-node3
5db5d9f4a1d2   redis:6.0.8   "docker-entrypoint.s…"   47 hours ago   Up 4 hours             redis-node2
7695b8a6290c   redis:6.0.8   "docker-entrypoint.s…"   47 hours ago   Up 4 hours             redis-node1
[root@CentOS7_001 ~]# docker stop redis-node1
redis-node1
[root@CentOS7_001 ~]# docker ps
CONTAINER ID   IMAGE         COMMAND                   CREATED        STATUS       PORTS     NAMES
b69b4d4562ab   redis:6.0.8   "docker-entrypoint.s…"   47 hours ago   Up 4 hours             redis-node6
51cf2c0eb2bf   redis:6.0.8   "docker-entrypoint.s…"   47 hours ago   Up 4 hours             redis-node5
c00847c63342   redis:6.0.8   "docker-entrypoint.s…"   47 hours ago   Up 4 hours             redis-node4
4c9337895da0   redis:6.0.8   "docker-entrypoint.s…"   47 hours ago   Up 4 hours             redis-node3
5db5d9f4a1d2   redis:6.0.8   "docker-entrypoint.s…"   47 hours ago   Up 4 hours             redis-node2
[root@CentOS7_001 ~]# docker exec -it 5db5d9f4a1d2 /bin/bash
root@CentOS7_001:/data# redis-cli -p 6382 -c
127.0.0.1:6382> cluster nodes
34fee68e4f9828340aea9dc71a2c2091981002ee 192.168.211.122:6383@16383 master - 0 1686896764000 3 connected 10923-16383
36877ecd044077c08ff640c14eb60698fada9e5a 192.168.211.122:6381@16381 master,fail - 1686896713157 1686896710000 1 disconnected
16567fbe877b25e90f97d9b2af76ef80ca3427d8 192.168.211.122:6386@16386 slave 6af5921ced502eae0ae93c991e1cafcee595e237 0 1686896765526 2 connected
07bd7bc35e2a88624855481ee24ff89d8cfcf759 192.168.211.122:6384@16384 slave 34fee68e4f9828340aea9dc71a2c2091981002ee 0 1686896764520 3 connected
ac38a3672a052a72855157d03e5c1bf9b73c717e 192.168.211.122:6385@16385 master - 0 1686896766530 7 connected 0-5460
6af5921ced502eae0ae93c991e1cafcee595e237 192.168.211.122:6382@16382 myself,master - 0 1686896765000 2 connected 5461-10922
127.0.0.1:6382> exit

恢复node1节点;观察

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

[root@CentOS7_001 ~]# docker start redis-node1
redis-node1
[root@CentOS7_001 ~]# docker ps
CONTAINER ID   IMAGE         COMMAND                   CREATED      STATUS         PORTS     NAMES
b69b4d4562ab   redis:6.0.8   "docker-entrypoint.s…"   2 days ago   Up 5 hours               redis-node6
51cf2c0eb2bf   redis:6.0.8   "docker-entrypoint.s…"   2 days ago   Up 5 hours               redis-node5
c00847c63342   redis:6.0.8   "docker-entrypoint.s…"   2 days ago   Up 5 hours               redis-node4
4c9337895da0   redis:6.0.8   "docker-entrypoint.s…"   2 days ago   Up 5 hours               redis-node3
5db5d9f4a1d2   redis:6.0.8   "docker-entrypoint.s…"   2 days ago   Up 5 hours               redis-node2
7695b8a6290c   redis:6.0.8   "docker-entrypoint.s…"   2 days ago   Up 2 seconds             redis-node1
[root@CentOS7_001 ~]# docker exec -it redis-node1 /bin/bash
root@CentOS7_001:/data# redis-cli -p 6381 -c
127.0.0.1:6381> cluster nodes
36877ecd044077c08ff640c14eb60698fada9e5a 192.168.211.122:6381@16381 myself,slave ac38a3672a052a72855157d03e5c1bf9b73c717e 0 1686901124000 7 connected
6af5921ced502eae0ae93c991e1cafcee595e237 192.168.211.122:6382@16382 master - 0 1686901125589 2 connected 5461-10922
34fee68e4f9828340aea9dc71a2c2091981002ee 192.168.211.122:6383@16383 master - 0 1686901124583 3 connected 10923-16383
ac38a3672a052a72855157d03e5c1bf9b73c717e 192.168.211.122:6385@16385 master - 0 1686901124000 7 connected 0-5460
16567fbe877b25e90f97d9b2af76ef80ca3427d8 192.168.211.122:6386@16386 slave 6af5921ced502eae0ae93c991e1cafcee595e237 0 1686901122572 2 connected
07bd7bc35e2a88624855481ee24ff89d8cfcf759 192.168.211.122:6384@16384 slave 34fee68e4f9828340aea9dc71a2c2091981002ee 0 1686901121567 3 connected
127.0.0.1:6381>

### 重新进入node1容器;查看集群节点;
### 此时node1由停机前(master)转变成启动后(slave)

还原之前的3主3从

思路：启动停掉的6381,此时6381已成了slave;先停掉6381的master,再启动.此时就能恢复

主从扩容

新增两个服务节点对集群扩容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#### 新增6387,6388两个节点
docker run -d --name redis-node7 --privileged=true --net=host -v /mydata/redis/cluster/redis-node-7:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6387

docker run -d --name redis-node8 --privileged=true --net=host -v /mydata/redis/cluster/redis-node-8:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6388

[root@CentOS7_001 ~]# docker ps
CONTAINER ID   IMAGE         COMMAND                   CREATED          STATUS          PORTS     NAMES
a354a3b65af8   redis:6.0.8   "docker-entrypoint.s…"   2 seconds ago    Up 2 seconds              redis-node8
8428e85359ea   redis:6.0.8   "docker-entrypoint.s…"   27 seconds ago   Up 26 seconds             redis-node7
b69b4d4562ab   redis:6.0.8   "docker-entrypoint.s…"   7 days ago       Up 3 minutes              redis-node6
51cf2c0eb2bf   redis:6.0.8   "docker-entrypoint.s…"   7 days ago       Up 3 minutes              redis-node5
c00847c63342   redis:6.0.8   "docker-entrypoint.s…"   7 days ago       Up 3 minutes              redis-node4
4c9337895da0   redis:6.0.8   "docker-entrypoint.s…"   7 days ago       Up 3 minutes              redis-node3
5db5d9f4a1d2   redis:6.0.8   "docker-entrypoint.s…"   7 days ago       Up 3 minutes              redis-node2
7695b8a6290c   redis:6.0.8   "docker-entrypoint.s…"   7 days ago       Up 3 minutes              redis-node1


进入6387容器内部，进行主从节点挂载和内容空间分配

  • 进入容器实例内部，将实例节点加入集群

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    [root@CentOS7_001 ~]# docker exec -it 8428e85359ea /bin/bash
    root@CentOS7_001:/data#

    root@CentOS7_001:/data# redis-cli --cluster add-node 192.168.211.122:6387 192.168.211.122:6381
    >>> Adding node 192.168.211.122:6387 to cluster 192.168.211.122:6381
    >>> Performing Cluster Check (using node 192.168.211.122:6381)
    S: 36877ecd044077c08ff640c14eb60698fada9e5a 192.168.211.122:6381
       slots: (0 slots) slave
       replicates ac38a3672a052a72855157d03e5c1bf9b73c717e
    M: 34fee68e4f9828340aea9dc71a2c2091981002ee 192.168.211.122:6383
       slots:[10923-16383] (5461 slots) master
       1 additional replica(s)
    M: ac38a3672a052a72855157d03e5c1bf9b73c717e 192.168.211.122:6385
       slots:[0-5460] (5461 slots) master
       1 additional replica(s)
    S: 07bd7bc35e2a88624855481ee24ff89d8cfcf759 192.168.211.122:6384
       slots: (0 slots) slave
       replicates 34fee68e4f9828340aea9dc71a2c2091981002ee
    M: 6af5921ced502eae0ae93c991e1cafcee595e237 192.168.211.122:6382
       slots:[5461-10922] (5462 slots) master
       1 additional replica(s)
    S: 16567fbe877b25e90f97d9b2af76ef80ca3427d8 192.168.211.122:6386
       slots: (0 slots) slave
       replicates 6af5921ced502eae0ae93c991e1cafcee595e237
    [OK] All nodes agree about slots configuration.
    >>> Check for open slots...
    >>> Check slots coverage...
    [OK] All 16384 slots covered.
    >>> Send CLUSTER MEET to node 192.168.211.122:6387 to make it join the cluster.
    [OK] New node added correctly.
    root@CentOS7_001:/data#

    增加服务节点


  首次check集群

  • 重新分配槽号

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    root@CentOS7_001:/data# redis-cli --cluster reshard 192.168.211.122:6387
    >>> Performing Cluster Check (using node 192.168.211.122:6387)
    M: 73f0387ac7a8fde1429667c942d5bb4a6b11689e 192.168.211.122:6387
       slots: (0 slots) master
    S: 16567fbe877b25e90f97d9b2af76ef80ca3427d8 192.168.211.122:6386
       slots: (0 slots) slave
       replicates 6af5921ced502eae0ae93c991e1cafcee595e237
    S: 36877ecd044077c08ff640c14eb60698fada9e5a 192.168.211.122:6381
       slots: (0 slots) slave
       replicates ac38a3672a052a72855157d03e5c1bf9b73c717e
    M: 34fee68e4f9828340aea9dc71a2c2091981002ee 192.168.211.122:6383
       slots:[10923-16383] (5461 slots) master
       1 additional replica(s)
    S: 07bd7bc35e2a88624855481ee24ff89d8cfcf759 192.168.211.122:6384
       slots: (0 slots) slave
       replicates 34fee68e4f9828340aea9dc71a2c2091981002ee
    M: 6af5921ced502eae0ae93c991e1cafcee595e237 192.168.211.122:6382
       slots:[5461-10922] (5462 slots) master
       1 additional replica(s)
    M: ac38a3672a052a72855157d03e5c1bf9b73c717e 192.168.211.122:6385
       slots:[0-5460] (5461 slots) master
       1 additional replica(s)
    [OK] All nodes agree about slots configuration.
    >>> Check for open slots...
    >>> Check slots coverage...
    [OK] All 16384 slots covered.
    How many slots do you want to move (from 1 to 16384)? 4096
    What is the receiving node ID? 73f0387ac7a8fde1429667c942d5bb4a6b11689e
    Please enter all the source node IDs.
      Type 'all' to use all the nodes as source nodes for the hash slots.
      Type 'done' once you entered all the source nodes IDs.
    Source node #1: all

    Ready to move 4096 slots.
      Source nodes:
        M: 34fee68e4f9828340aea9dc71a2c2091981002ee 192.168.211.122:6383
           slots:[10923-16383] (5461 slots) master
           1 additional replica(s)
        M: 6af5921ced502eae0ae93c991e1cafcee595e237 192.168.211.122:6382
           slots:[5461-10922] (5462 slots) master
           1 additional replica(s)
        M: ac38a3672a052a72855157d03e5c1bf9b73c717e 192.168.211.122:6385
           slots:[0-5460] (5461 slots) master
           1 additional replica(s)
      Destination node:
        M: 73f0387ac7a8fde1429667c942d5bb4a6b11689e 192.168.211.122:6387
           slots: (0 slots) master
      Resharding plan:
        Moving slot 5461 from 6af5921ced502eae0ae93c991e1cafcee595e237
        Moving slot 5462 from 6af5921ced502eae0ae93c991e1cafcee595e237
        Moving slot 5463 from 6af5921ced502eae0ae93c991e1cafcee595e237
        Moving slot 5464 from 6af5921ced502eae0ae93c991e1cafcee595e237
        Moving slot 5465 from 6af5921ced502eae0ae93c991e1cafcee595e237
        Moving slot 5466 from 6af5921ced502eae0ae93c991e1cafcee595e237
        Moving slot 5467 from 6af5921ced502eae0ae93c991e1cafcee595e237
    ......
    哈希槽扩容重新分配

详细分配图

  第二次check集群

槽号分派说明

重新分配成本太高,所以前三家各自匀出来一部分，从6381，6382，6383三个主节点中分别匀出1364个坑位给新节点6387

  • 为6387主节点挂在6388从节点

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    #### 
    命令：redis-cli --cluster add-node ip:新slave端口 ip:新master端口 --cluster-slave --cluster-master-id 新主机节点ID


    root@CentOS7_001:/data# redis-cli --cluster add-node 192.168.211.122:6388 192.168.211.122:6387 --cluster-slave --cluster-                                   master-id 73f0387ac7a8fde1429667c942d5bb4a6b11689e
    >>> Adding node 192.168.211.122:6388 to cluster 192.168.211.122:6387
    >>> Performing Cluster Check (using node 192.168.211.122:6387)
    M: 73f0387ac7a8fde1429667c942d5bb4a6b11689e 192.168.211.122:6387
       slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
    S: 16567fbe877b25e90f97d9b2af76ef80ca3427d8 192.168.211.122:6386
       slots: (0 slots) slave
       replicates 6af5921ced502eae0ae93c991e1cafcee595e237
    S: 36877ecd044077c08ff640c14eb60698fada9e5a 192.168.211.122:6381
       slots: (0 slots) slave
       replicates ac38a3672a052a72855157d03e5c1bf9b73c717e
    M: 34fee68e4f9828340aea9dc71a2c2091981002ee 192.168.211.122:6383
       slots:[12288-16383] (4096 slots) master
       1 additional replica(s)
    S: 07bd7bc35e2a88624855481ee24ff89d8cfcf759 192.168.211.122:6384
       slots: (0 slots) slave
       replicates 34fee68e4f9828340aea9dc71a2c2091981002ee
    M: 6af5921ced502eae0ae93c991e1cafcee595e237 192.168.211.122:6382
       slots:[6827-10922] (4096 slots) master
       1 additional replica(s)
    M: ac38a3672a052a72855157d03e5c1bf9b73c717e 192.168.211.122:6385
       slots:[1365-5460] (4096 slots) master
       1 additional replica(s)
    [OK] All nodes agree about slots configuration.
    >>> Check for open slots...
    >>> Check slots coverage...
    [OK] All 16384 slots covered.
    >>> Send CLUSTER MEET to node 192.168.211.122:6388 to make it join the cluster.
    Waiting for the cluster to join

    >>> Configure node as replica of 192.168.211.122:6387.
    [OK] New node added correctly.

    扩容主节点挂载从节点


  第三次check集群

  • 扩容完成

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    root@CentOS7_001:/data# redis-cli --cluster check 192.168.211.122:6387
    192.168.211.122:6387 (73f0387a...) -> 0 keys | 4096 slots | 1 slaves.
    192.168.211.122:6383 (34fee68e...) -> 1 keys | 4096 slots | 1 slaves.
    192.168.211.122:6382 (6af5921c...) -> 0 keys | 4096 slots | 1 slaves.
    192.168.211.122:6385 (ac38a367...) -> 0 keys | 4096 slots | 1 slaves.
    [OK] 1 keys in 4 masters.
    0.00 keys per slot on average.
    >>> Performing Cluster Check (using node 192.168.211.122:6387)
    M: 73f0387ac7a8fde1429667c942d5bb4a6b11689e 192.168.211.122:6387
       slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
       1 additional replica(s)
    S: 16567fbe877b25e90f97d9b2af76ef80ca3427d8 192.168.211.122:6386
       slots: (0 slots) slave
       replicates 6af5921ced502eae0ae93c991e1cafcee595e237
    S: 36877ecd044077c08ff640c14eb60698fada9e5a 192.168.211.122:6381
       slots: (0 slots) slave
       replicates ac38a3672a052a72855157d03e5c1bf9b73c717e
    S: 6bf8e9d337e27154408acbd2a6a9677bdde939da 192.168.211.122:6388
       slots: (0 slots) slave
       replicates 73f0387ac7a8fde1429667c942d5bb4a6b11689e
    M: 34fee68e4f9828340aea9dc71a2c2091981002ee 192.168.211.122:6383
       slots:[12288-16383] (4096 slots) master
       1 additional replica(s)
    S: 07bd7bc35e2a88624855481ee24ff89d8cfcf759 192.168.211.122:6384
       slots: (0 slots) slave
       replicates 34fee68e4f9828340aea9dc71a2c2091981002ee
    M: 6af5921ced502eae0ae93c991e1cafcee595e237 192.168.211.122:6382
       slots:[6827-10922] (4096 slots) master
       1 additional replica(s)
    M: ac38a3672a052a72855157d03e5c1bf9b73c717e 192.168.211.122:6385
       slots:[1365-5460] (4096 slots) master
       1 additional replica(s)
    [OK] All nodes agree about slots configuration.
    >>> Check for open slots...
    >>> Check slots coverage...
    [OK] All 16384 slots covered.
    root@CentOS7_001:/data#

主从缩容

能扩容就要能缩容,减少服务器性能浪费

  • 删除6388从节点

    1
    2
    3
    4
    5
    6
    7
    8
    9
    #### 命令：redis-cli --cluster del-node ip:从机端口 从机6388节点ID


    root@CentOS7_001:/data# redis-cli --cluster del-node 192.168.211.122:6388 6bf8e9d337e27154408acbd2a6a9677bdde939da
    >>> Removing node 6bf8e9d337e27154408acbd2a6a9677bdde939da from cluster 192.168.211.122:6388
    >>> Sending CLUSTER FORGET messages to the cluster...
    >>> Sending CLUSTER RESET SOFT to the deleted node.
    root@CentOS7_001:/data#

  • 重新分配6387master服务槽号,并删除6387节点

    1
    2
    3
    4
    5
    6
    7
    #### 命令： redis-cli --cluster reshard 192.168.211.122:6387

    redis-cli --cluster del-node 192.168.211.122:6387 73f0387ac7a8fde1429667c942d5bb4a6b11689e
    >>> Removing node 73f0387ac7a8fde1429667c942d5bb4a6b11689e from cluster 192.168.211.122:6387
    >>> Sending CLUSTER FORGET messages to the cluster...
    >>> Sending CLUSTER RESET SOFT to the deleted node.

    哈希槽缩容重新分配

  • 删除容器实例
  • 缩容完成
此页目录
Docker部署redis集群
  1. 分布式存储案例
  2. cluster(集群)模式-docker版
    1. 经典问题
  3. Redis三主三从集群配置
    1. 前置条件
    2. 创建redis实例
    3. 构建主从关系
    4. 查看集群状态
    5. 读写存储
    6. 查看集群信息
    7. 容错切换迁移
    8. 主从扩容
    9. 主从缩容