主从复制问题

  在了解sentinel前,先说说redis主从复制中的问题.假设主从架构,master:写,slave1和slave2:读.当master出现问题.客户端读取数据没有问题,写入数据出现问题.一般的解决方式就是选取另一个slave做master.其他slave主从复制新的master.这是需要手动去处理的.我们也不知道什么会出现问题,不可能即时的去修改.

  而redis推出了sentinel很好的解决上述问题

Sentinel架构

架构介绍

  首先依然是主从复制架构master.slave1.slave2.然后还要许多的sentinel节点.可以当成redis的进程,不会存储数据.只是完成对redis进行故障判断和故障转移的功能,并且会通知客户端变更后的可链接的地址.

  多个sentinel可以保证对redis故障判断的公平性.同时高可用,当一个sentinel挂了.sentinel节点也是完整可用的.

  对于客户端链操作获取数据再也不用直接通过redis获取.而是通过sentinel获取redis信息.sentinel会对redis的主从架构进行监控.知道谁是master,谁是slave.

故障转移介绍

  sentinel的故障转移和刚开始介绍的主从复制手动转移原理是一样的.但不需要去手动的管理.可以监控多套redis主从架构.一套sentinel可以监控多个redis主从架构.

  1. 如果master和slave的链接中断.
  2. 多个sentinel发现并确认master有问题.
  3. sentinel节点会选举其中一个sentinel作为领导.
  4. sentinel领导将其中一个slave作为master
  5. 通知其余slave复制新的master
  6. 告知客户端新的master.防止访问之前宕机的master
  7. 当老master复活.sentinel将老的master变成新的slave

Sentinel使用

配置主从架构

# 1主,3从
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:3

查看sentenl配置

# redis本来就有sentinel配置.无需去下载
cp sentinel.conf config/sentinel.conf
# 查看默认配置
cat config/sentinel.conf | grep -v '#' | grep -v '^$'
port 26379
daemonize no
pidfile /var/run/redis-sentinel.pid
logfile ""
dir /tmp
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes

配置说明

# sentinel 端口
port 26379
-- R_name 为redis主从节点名,用于区分不同架构
-- 主节点ip 端口号 
-- sentine_num几个节点发现有问题后开始处理问题
sentinel monitor R_name host port sentine_num
# 3万秒ping不通相当于有问题
sentinel down-after-milliseconds mymaster 30000 
# 代表每次复制1个slave。减轻master压力
sentinel parallel-syncs mymaster 1 
# 故障转移时间
sentinel failover-timeout mymaster 180000

修改配置

port 26379
daemonize yes
pidfile /var/run/redis-sentinel.pid
logfile "sentinel_26379.log"
dir /data/log/redis/data/
sentinel monitor mymaster 192.168.128.131 6379 2
sentinel down-after-milliseconds mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes

启动

# 启动
[root@bogon config]# redis-sentinel sentinel_26379.conf
# 查看进程
[root@bogon config]# lsof -i:26379
COMMAND     PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
redis-sen 18577 root    6u  IPv6 5998962      0t0  TCP *:26379 (LISTEN)
redis-sen 18577 root    7u  IPv4 5998963      0t0  TCP *:26379 (LISTEN)
# 链接
[root@bogon config]# redis-cli -p 26379
127.0.0.1:26379> set angel qvbilam
(error) ERR unknown command `set`, with args beginning with: `angel`, `qvbilam`,
# 查看信息
127.0.0.1:26379> info
# Sentinel
sentinel_masters:1
sentinel_tilt:1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.128.131:6379,slaves=3,sentinels=1
# 当启动成功后会发现配置文件发生改变
[root@bogon config]# cat sentinel_26379.conf
port 26379
daemonize yes
pidfile "/var/run/redis-sentinel.pid"
logfile "sentinel_26379.log"
dir "/data/log/redis/data"
sentinel myid 27f5ebf905c1f1c5f99ec4982c7711be6487575b
sentinel deny-scripts-reconfig yes
sentinel monitor mymaster 192.168.128.131 6379 2
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
# Generated by CONFIG REWRITE
protected-mode no
sentinel known-replica mymaster 192.168.128.134 6379
sentinel known-replica mymaster 192.168.128.133 6379
sentinel known-replica mymaster 192.168.128.132 6379
sentinel current-epoch 0
# 接着在配置俩个sentenl,在132和133上.
# 在master的sentel查看info
master0:name=mymaster,status=ok,address=192.168.128.131:6379,slaves=3,sentinels=3
# 发现有主从架构有3个从节点.一共有3个sentinel.

Client使用

# 初始化composer.
# 一路回车当有选项Author [, n to skip]就n
composer init
# 下载自动加载类
composer dump-autoload

  当通过predis链接redis的时候可能出现下列错误

PHP Fatal error:  Uncaught Predis\ClientException: No sentinel server available for autodiscovery.
# 解决办法将链接地址改成tcp链接方式.即ip都加上协议
$sentinel = [
    'tcp://192.168.128.131:26379',
    'tcp://192.168.128.132:26379',
    'tcp://192.168.128.133:26379',
];
# 再次启动出现
PHP Fatal error:  Uncaught Predis\ClientException: No sentinel server available for autodiscovery.
# 可能是端口没开.无法访问造成的
Fatal error: Uncaught Predis\Response\ServerException: ERR No such master with that name
# redis集群名不存在.设置错了..

开启端口

vim  /etc/sysconfig/iptables
# 添加
-A INPUT -p tcp -m state --state NEW -m tcp --dport 26379 -j ACCEPT
# 重启iptabels
service iptables restart
# 重启防火墙,建议都执行一下
// 即时生效,重启生效
service iptables start
// 重启后生效
chkconfig iptables off
chkconfig iptables on

demo代码

<?php

require 'vendor/autoload.php';
$sentinel = [
    'tcp://192.168.128.131:26379',
    'tcp://192.168.128.132:26379',
    'tcp://192.168.128.133:26379',
];
$options = [
    'replication' => 'sentinel',
    // 这里一定要和sentinel中的redis集群配置一样
    'service' => 'mymaster'
];
$redis = new Predis\Client($sentinel,$options);
$redis->set('angel','qvbilam');
$res = $redis->get('angel');
echo $res . PHP_EOL;

// 结果
[root@localhost test]# php sentinel.php
qvbilam

故障模拟

修改demo部分代码

$redis = new Predis\Client($sentinel,$options);
$i=0;
while(true){
    try{
        $redis->set('angel','qvbilam');
        $res = $redis->get('angel');
        if($i % 100 == 0){
            $content = $res . PHP_EOL;
            $file = 'sentinel.log';
            file_put_contents($file,$content,FILE_APPEND);
        }  
    }catch(Exception $e){
        file_put_contents($file,'错误' . PHP_EOL,FILE_APPEND);
    }    
}

挂掉主节点

# 查看进程id
[root@bogon config]# redis-cli info server | grep id
redis_build_id:b5f54c998fadfce4
process_id:48722
run_id:2003bbe3ea129f33a7b055b0937c67fdcfd7c9e5
# 启动php脚本,挂掉主节点
kill -9 48722
# 查看后10行的日志文件
[root@localhost test]# tail -f -n 10 sentinel.log
qvbilam
...
# 全是输出结果.查看是否出现错误
[root@localhost test]# cat sentinel.log | grep -v 'qvbilam'
错误
# 出现一次的样子.接着sentinel执行错误转移就继续能用了

redis日志

# 我之前做过一次没记录的测试.134变为master,131为slave
# master日志在最后还在执行主从复制.
103850:C 30 Jun 2019 04:02:15.032 * DB saved on disk
103850:C 30 Jun 2019 04:02:15.035 * RDB: 0 MB of memory used by copy-on-write
48722:M 30 Jun 2019 04:02:15.110 * Background saving terminated with success
48722:M 30 Jun 2019 04:02:15.110 * Synchronization with replica 192.168.128.131:6379 succeeded
# slave升级master日志
98403:S 30 Jun 2019 04:09:55.899 # Connection with master lost.
98403:S 30 Jun 2019 04:09:55.899 * Caching the disconnected master state.
98403:S 30 Jun 2019 04:09:56.067 * Connecting to MASTER 192.168.128.134:6379
98403:S 30 Jun 2019 04:09:56.067 * MASTER <-> REPLICA sync started
98403:S 30 Jun 2019 04:09:56.068 # Error condition on socket for SYNC: Connection refused
# 先是发现大量的master链接不成功

# 在30秒的时候,收到希望它成为master并收到132,133的请求
98403:S 30 Jun 2019 04:10:26.240 * MASTER <-> REPLICA sync started
98403:S 30 Jun 2019 04:10:26.241 # Error condition on socket for SYNC: Connection refused
98403:M 30 Jun 2019 04:10:26.344 # Setting secondary replication ID to 241d78cf80a09a5c28388005232b64dfff7afc63, valid up to offset: 15588679. New replication ID is d59db800434a4ec913f7d74671c5350ecff19799
98403:M 30 Jun 2019 04:10:26.344 * Discarding previously cached master state.
98403:M 30 Jun 2019 04:10:26.344 * MASTER MODE enabled (user request from 'id=10 addr=192.168.128.131:59008 fd=9 name=sentinel-27f5ebf9-cmd age=492 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=140 qbuf-free=32628 obl=36 oll=0 omem=0 events=r cmd=exec')
98403:M 30 Jun 2019 04:10:26.344 # CONFIG REWRITE executed with success.
98403:M 30 Jun 2019 04:10:27.036 * Replica 192.168.128.132:6379 asks for synchronization
98403:M 30 Jun 2019 04:10:27.036 * Partial resynchronization request from 192.168.128.132:6379 accepted. Sending 26102 bytes of backlog starting from offset 15588679.
98403:M 30 Jun 2019 04:10:27.748 * Replica 192.168.128.133:6379 asks for synchronization
98403:M 30 Jun 2019 04:10:27.748 * Partial resynchronization request from 192.168.128.133:6379 accepted. Sending 173854 bytes of backlog starting from offset 15588679.


# 其他slave
78196:S 30 Jun 2019 04:10:27.277 * Connecting to MASTER 192.168.128.134:6379
78196:S 30 Jun 2019 04:10:27.277 * MASTER <-> REPLICA sync started
78196:S 30 Jun 2019 04:10:27.279 # Error condition on socket for SYNC: Connection refused
78196:S 30 Jun 2019 04:10:28.278 * REPLICAOF 192.168.128.131:6379 enabled (user request from 'id=1385370 addr=192.168.128.131:51276 fd=9 name=sentinel-27f5ebf9-cmd age=667 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=155 qbuf-free=32613 obl=36 oll=0 omem=0 events=r cmd=exec')
78196:S 30 Jun 2019 04:10:28.278 # CONFIG REWRITE executed with success.
78196:S 30 Jun 2019 04:10:28.308 * Connecting to MASTER 192.168.128.131:6379
78196:S 30 Jun 2019 04:10:28.309 * MASTER <-> REPLICA sync started
78196:S 30 Jun 2019 04:10:28.320 * Non blocking connect for SYNC fired the event.
78196:S 30 Jun 2019 04:10:28.322 * Master replied to PING, replication can continue...
78196:S 30 Jun 2019 04:10:28.330 * Trying a partial resynchronization (request 241d78cf80a09a5c28388005232b64dfff7afc63:15588679).
78196:S 30 Jun 2019 04:10:28.341 * Successful partial resynchronization with master.
78196:S 30 Jun 2019 04:10:28.341 # Master replication ID changed to d59db800434a4ec913f7d74671c5350ecff19799
78196:S 30 Jun 2019 04:10:28.341 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.
# 链接131成功

sentinel日志

# 发现节点可能出现问题
-sdown slave 192.168.128.131:6379 192.168.128.131 6379 @ mymaster 192.168.128.134 6379
# 一条sentinel确认
+sdown master mymaster 192.168.128.134 6379
# 两条确认,满足失联条件
+odown master mymaster 192.168.128.134 6379 #quorum 2/2
# sentinel配置权加2,此台尝试去做故障转移
+new-epoch 2
+try-failover master mymaster 192.168.128.134 6379
# 此台sentinel尝试做领导者
+vote-for-leader 27f5ebf905c1f1c5f99ec4982c7711be6487575b 2
# 另外两台sentinel投票2给它,选举领导
49704e3f60272bd65fd191231b2e606b8cd7c6db voted for 27f5ebf905c1f1c5f99ec4982c7711be6487575b 2
b012a3103d725142216d35896414b9f45ada1818 voted for 27f5ebf905c1f1c5f99ec4982c7711be6487575b 2
# sentinel选取集群中的新master.131替换134
selected-slave slave 192.168.128.131:6379 192.168.128.131 6379 @ mymaster 192.168.128.134 6379
# 给131发送slaveof-noneone.成为master
failover-state-send-slaveof-noone slave 192.168.128.131:6379 192.168.128.131 6379 @ mymaster 192.168.128.134 6379
# 等待131晋升为master
failover-state-wait-promotion slave 192.168.128.131:6379 192.168.128.131 6379 @ mymaster 192.168.128.134 6379
# 其他slave对新master进行复制
slave-reconf-sent slave 192.168.128.132:6379 192.168.128.132 6379 @ mymaster 192.168.128.134 6379
# 再次对旧master做下线标识,为了当天重启去当slave
-odown master mymaster 192.168.128.134 6379
# 目前完整的master/slave信息
+slave slave 192.168.128.134:6379 192.168.128.134 6379 @ mymaster 192.168.128.131 6379

# 而其他的sentinel日志就是发现问题
# 投票选举领导者
# 改变master/slave的信息

深入理解

内部定时任务

  在sentinel内部中有三个定时任务用于监控集群.

每10秒所有sentinel对集群执行info获取信息.

  通过对master获取信息可以发现slave节点.在sentinel只设置了master,而没设置slave节点.却能发现slave的原因.内部记录从节点信息.

  再对slave做info,可以看到slave下是否还要slave,以及节点信息的变化.

每2秒所有sentinel通过master的channel交换信息

  在master节点上会有一个发布订阅的频道(__sentinel__:helo),用于让sentinel节点进行信息交换.原理是,一个sentinel发送信息,其他sentinel订阅了都能收到发布的信息进行对比,达成共识.

所有sentinel会对其他sentinel以及redis进行ping

  对redis进行检测.以及错误判断.

主观/客观 下线

主观下线:单台sentinel认为redis节点失效.属于个人偏见.

客观下线: 所有(或者满足quornum)认为redis节点失效.属于达成共识.

# sentinel的配置
# 当quorum达到多少进行客观下线
sentinel monitor <masterName> <ip> <port> <quornum>
# 当超过timeout毫秒进行主观下线判断
sentinel down-after-milliseconds <masterName> <timeout>

  当一台sentinel出现主观下线,会将信息发送给其他sentinel.(对其他sentinel执行sentinel is master-down-by-addr,询问其他sentienl是否也认为有问题).达成共识后就可以做客观下线.接下来就开始做故障转移操作.

领导者选举

  在sentinel故障转移的过程中,只会有一个sentinel去操作redis节点.当有很多个sentinel.会选举出一个领导者去执行故障转移.通过sentinel is-master-down-by-addr命令成为领导者.这个命令一是交换对mater节点失败判定,在一个就是做领导者的选举.

  每个sentinel做完主观下线后会对其他sentinel发送上述命令,并希望自己成为领导者.其他sentinel对此进行判断是否同意.当sentinel发现自己的票数超过sentinel集群半数且达到了quornum将成为领导者.如果多个sentinel都成为领导者,那就等待一段时间重新进行一轮选举.

故障转移

故障转移步骤

  1. 从slave中选取其中一个合适的节点作为新的master.
  2. 对选中的slave执行slave no one,使它成为新的master.
  3. 向剩余的slave发送命令,让他们改变master.
  4. 将原来的master标记为slave.
  5. 时刻保持对原来master的监听.当恢复后使其成为slave.

合适的节点

  1. 配置中的slave-priority较高的(slave优先级高的).配置中默认没有.如果存在就返回该节点.
  2. 选择偏移量更大的节点.说明和master的数据更接近.如果存在就返回该节点.
  3. run_id最小的节点(启动时间最早).
Last modification：February 18th, 2020 at 10:17 pm
© 允许规范转载