Linux下 Redis主从架构持久化操作详述

218次阅读

共计 13593 个字符，预计需要花费 34 分钟才能阅读完成。

Redis 主从架构持久化存在一个问题，即前次测试的结论，持久化需要配置在主实例上才能跨越实例保证数据不丢失，这样以来主实例在持久化数据到硬盘的过程中，势必会造成磁盘的 I / O 等待，经过实际测试，这个持久化写硬盘的过程给应用程序带来的影响无法忍受；因而在大多数场景下，会考虑把持久化配置在从实例上，当主实例宕机后，通过手动或者自动的方式将从实例提升为主实例，继续提供服务！当主实例恢复后，先从原从实例上同步数据，同步完成后再恢复到原始的主从状态！要实现这种的要求，需要有 keepalive 的配合，一方面 keepalive 提供了 VIP，可以避免修改应用程序连接，同时 redis 实例的配置文件监听部分也需要修改为全网监听；另一方面 keepalive 定时调度脚本来监控主从实例的状态，根据具体情况进行切换！本文将重点介绍下使用 keepalive 实现 redis 主从自动 failover！

环境介绍
操作系统版本均为：rhel5.4 64bit
redis 版本：2.6.4
redis 实例端口均为：6379
redis 实例密码均为：123
VIP：192.168.1.120
主实例为 server11(192.168.1.112)
从实例为 server12(192.168.1.113，开启快照持久化)

一：安装 keepalive 软件,server11 安装完成后直接 scp 至 server12 上即可

[root@server11 ~]# wget http://keepalived.org/software/keepalived-1.1.19.tar.gz
[root@server11 ~]# tar -zxvf ../tarbag/keepalived-1.1.19.tar.gz
[root@server11 ~]# cd keepalived-1.1.19/
[root@server11 ~]# ./configure –prefix=/usr/local/keepalived && make && make install

二：配置主节点 server11 配置文件

[root@server11 ~]# cat /usr/local/keepalived/etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
router_id LVS_DEVEL
}

vrrp_script Monitor_redis {
script “/usr/local/scripts/redis_monitor.sh”
interval 2
weight 2
}

vrrp_instance VI_1{
state MASTER
interface eth0
virtual_router_id 51
mcast_src_ip 192.168.1.112
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass password_123
}
track_script {
Monitor_redis
}
virtual_ipaddress {
192.168.1.120
}
notify_fault /usr/local/scripts/redis_fault.sh
notify_stop /usr/local/scripts/redis_stop.sh

}

三：配置从节点 server12 配置文件

[root@server12 ~]# cat /usr/local/keepalived/etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
router_id LVS_DEVEL
}

vrrp_script Monitor_redis {
script “/usr/local/scripts/redis_monitor.sh”
interval 2
weight 2
}

vrrp_instance VI_1{
state BACKUP
interface eth0
virtual_router_id 51
mcast_src_ip 192.168.1.113
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass password_123
}
track_script {
Monitor_redis
}
virtual_ipaddress {
192.168.1.120
}
notify_master /usr/local/scripts/redis_master.sh
notify_backup /usr/local/scripts/redis_backup.sh
notify_fault /usr/local/scripts/redis_fault.sh
notify_stop /usr/local/scripts/redis_stop.sh

}

四：准备相关的脚本，主从实例上都需要存在这些脚本，同时注意脚本需要由可执行权限

[root@server11 ~]# cat /usr/local/scripts/redis_monitor.sh
#!/bin/bash
ALIVE=$(/usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123 PING)

if [“$ALIVE” == “PONG”]; then
echo $ALIVE
exit 0
else
echo $ALIVE
killall -9 keepalived
service network restart
exit 1
fi

[root@server11 ~]# sh /usr/local/scripts/redis_monitor.sh
PONG

[root@server11 ~]# cat /usr/local/scripts/redis_master.sh
#!/bin/bash
REDISCLI=”/usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123″
LOGFILE=”/usr/local/redis2/var/keepalived-redis-state.log”

echo “[master]” >> $LOGFILE
date >> $LOGFILE
echo “Being master….” >> $LOGFILE 2>&1
echo “Run SLAVEOF cmd …” >> $LOGFILE
$REDISCLI SLAVEOF 192.168.1.113 6379 >> $LOGFILE 2>&1
sleep 10
echo “Run SLAVEOF NO ONE cmd …” >> $LOGFILE
$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1

[root@server11 ~]# cat /usr/local/scripts/redis_backup.sh
#!/bin/bash
REDISCLI=”/usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123″
LOGFILE=”/usr/local/redis2/var/keepalived-redis-state.log”

echo “[backup]” >> $LOGFILE
date >> $LOGFILE
echo “Being slave….” >> $LOGFILE 2>&1
sleep 15
echo “Run SLAVEOF cmd …” >> $LOGFILE
$REDISCLI SLAVEOF 192.168.1.113 6379 >> $LOGFILE 2>&1

[root@server11 ~]# cat /usr/local/scripts/redis_stop.sh
#!/bin/bash
LOGFILE=”/usr/local/redis2/var/keepalived-redis-state.log”
echo “[stop]” >> $LOGFILE
date >> $LOGFILE
[root@server11 ~]# cat /usr/local/scripts/redis_fault.sh
#!/bin/bash
LOGFILE=”/usr/local/redis2/var/keepalived-redis-state.log”
echo “[fault]” >> $LOGFILE
date >> $LOGFILE

五：主从实例分别启动 keepalive 进程，测试 VIP 是否正常(这里就要修改 redis 配置文件的监听地址为 0.0.0.0)

[root@server11 ~]# /usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf
[root@server11 ~]# tail -f /var/log/messages
Dec 12 09:25:49 server11 Keepalived_healthcheckers[7710]: Configuration is using : 5499 Bytes
Dec 12 09:25:49 server11 Keepalived_healthcheckers[7710]: Using LinkWatch kernel netlink reflector…
Dec 12 09:25:49 server11 Keepalived_vrrp[7712]: VRRP sockpool: [ifindex(2), proto(112), fd(12,13)]
Dec 12 09:25:49 server11 Keepalived_vrrp[7712]: VRRP_Script(Monitor_redis) succeeded
Dec 12 09:25:50 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Transition to MASTER STATE
Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Entering MASTER STATE
Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) setting protocol VIPs.
Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120
Dec 12 09:25:51 server11 avahi-daemon[4519]: Registering new address record for 192.168.1.120 on eth0.
Dec 12 09:25:51 server11 Keepalived_healthcheckers[7710]: Netlink reflector reports IP 192.168.1.120 added
Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: Netlink reflector reports IP 192.168.1.120 added
Dec 12 09:25:56 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120

[root@server11 ~]# ip a |grep 192
inet 192.168.1.112/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.120/32 scope global eth0

[root@server12 ~]# /usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf
[root@server12 ~]# tail -f /var/log/messages
Dec 12 09:26:55 server12 Keepalived_healthcheckers[3106]: Configuration is using : 5595 Bytes
Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Entering BACKUP STATE
Dec 12 09:26:55 server12 Keepalived_healthcheckers[3106]: Using LinkWatch kernel netlink reflector…
Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP sockpool: [ifindex(2), proto(112), fd(12,13)]
Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP_Script(Monitor_redis) succeeded

[root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |grep -A 3 ‘Replication’
# Replication
role:master
connected_slaves:1
slave0:192.168.1.113,6379,online

六：主实例写入测试数据，该脚本原则上会写入 25 条测试数据，不过由于未优化 redis 默认并发数，会导致一些写入请求失败，最终功写入 231839 条测试数据，占内存总大小为 25M 左右，写入过程中可以观察主从实例的持久化文件变化情况，主实例的持久化文件维持在 30k，从实例的则不断的扩展！

[root@server11 ~]# cat test.sh
#!/bin/bash
REDISCLI=”/usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 -n 1 SET”
ID=1
while(($ID<50001))
do
INSTANCE_NAME=”i-2-$ID-VM”
UUID=`cat /proc/sys/kernel/random/uuid`
PRIVATE_IP_ADDRESS=10.`echo “$RANDOM % 255 + 1” | bc`.`echo “$RANDOM % 255 + 1” | bc`.`echo

“$RANDOM % 255 + 1” | bc`\
CREATED=`date “+%Y-%m-%d %H:%M:%S”`
$REDISCLI vm_instance:$ID:instance_name “$INSTANCE_NAME”
$REDISCLI vm_instance:$ID:uuid “$UUID”
$REDISCLI vm_instance:$ID:private_ip_address “$PRIVATE_IP_ADDRESS”
$REDISCLI vm_instance:$ID:created “$CREATED”
$REDISCLI vm_instance:$INSTANCE_NAME:id “$ID”
ID=$(($ID+1))
done

[root@server11 ~]# sh test.sh
[root@server11 redis2]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep

‘used_memory_peak_human|db1:keys’
used_memory_peak_human:24.98M
db1:keys=231839,expires=0

七：模拟主实例故障，观察日志输出，验证从实例是否能成功接管 VIP，同时将实例变成读写模式

[root@server11 ~]# killall -9 redis-server
[root@server11 ~]# ip a |grep 192
inet 192.168.1.112/24 brd 192.168.1.255 scope global eth0

[root@server11 ~]# ps -ef |grep redis
root 15886 6458 0 09:49 pts/0 00:00:00 grep redis
[root@server11 ~]# ps -ef |grep keep
root 16029 6458 0 09:49 pts/0 00:00:00 grep keep

[root@server12 ~]# tail -f /usr/local/redis2/var/keepalived-redis-state.log
[master]
Wed Dec 12 09:48:52 CST 2012
Being master….
Run SLAVEOF cmd …
OK Already connected to specified master
Run SLAVEOF NO ONE cmd …
OK

[root@server12 ~]# tail -f /var/log/messages
Dec 12 09:48:51 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Transition to MASTER STATE
Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Entering MASTER STATE
Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) setting protocol VIPs.
Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120
Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: Netlink reflector reports IP 192.168.1.120 added
Dec 12 09:48:52 server12 avahi-daemon[2921]: Registering new address record for 192.168.1.120 on eth0.
Dec 12 09:48:52 server12 Keepalived_healthcheckers[3106]: Netlink reflector reports IP 192.168.1.120 added
Dec 12 09:48:57 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120

[root@server12 ~]# ip a |grep 192
inet 192.168.1.113/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.120/32 scope global eth0

[root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |grep -A 3 ‘Replication’
# Replication
role:master
connected_slaves:0

[root@server12 ~]# sh test.sh
[root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep

‘used_memory_peak_human|db1:keys’
used_memory_peak_human:26.78M
db1:keys=249925,expires=0

九: 主实例角色的恢复过程，使用 shell 脚本自动恢复

[root@server11 ~]# ssh-keygen
[root@server11 ~]# cd .ssh/
[root@server11 .ssh]# ssh-copy-id -i id_rsa.pub root@192.168.1.113
[root@server11 ~]# cat /usr/local/scripts/recover_mastart.sh
#!/bin/sh
ALIVE=$(/usr/local/redis2/bin/redis-cli -h 192.168.1.113 -p 6379 -a 123 PING)
MDB=/usr/local/redis2/master_dump.rdb
SDB=/usr/local/redis2/slave_dump.rdb

if [“$ALIVE” == “PONG”]; then
echo $ALIVE
scp root@192.168.1.113:$SDB $MDB
else
echo $ALIVE
exit 1
fi

/usr/local/redis2/bin/redis-server /usr/local/redis2/etc/redis.conf
/usr/local/keepalived/sbin/keepalived -D -f

/usr/local/keepalived/etc/keepalived/keepalived.conf

[root@server11 ~]# chmod +x /usr/local/scripts/recover_mastart.sh
[root@server11 ~]# sh /usr/local/scripts/recover_mastart.sh

十：验证数据完整性和主从角色恢复情况

[root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep ‘used_memory_peak_human|db1:keys’
used_memory_peak_human:26.78M
db1:keys=249925,expires=0

[root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |grep -A 3 ‘Replication’
# Replication
role:master
connected_slaves:1
slave0:192.168.1.113,6379,online

[root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 info |grep -A 3 ‘Replication’
# Replication
role:slave
master_host:192.168.1.112
master_port:6379

[root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep ‘used_memory_peak_human|db1:keys’
used_memory_peak_human:26.78M
db1:keys=249925,expires=0

主实例 keepalive 日志：
[root@server11 ~]# tail -f /var/log/messages
Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]
Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Script(Monitor_redis) succeeded
Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Transition to MASTER STATE
Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Received higher prio advert
Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Entering BACKUP STATE
Dec 12 10:08:15 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) forcing a new MASTER election
Dec 12 10:08:16 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Transition to MASTER STATE
Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Entering MASTER STATE
Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) setting protocol VIPs.
Dec 12 10:08:17 server11 Keepalived_healthcheckers[20230]: Netlink reflector reports IP 192.168.1.120 added
Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120
Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: Netlink reflector reports IP 192.168.1.120 added
Dec 12 10:08:17 server11 avahi-daemon[4519]: Registering new address record for 192.168.1.120 on eth0.

[root@server11 ~]# ip a |grep 192
inet 192.168.1.112/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.120/32 scope global eth0

从实例 keepalive 日志：
[root@server12 ~]# tail -f /var/log/messages
Dec 12 09:56:01 server12 last message repeated 4 times
Dec 12 10:08:13 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Received lower prio advert, forcing new election
Dec 12 10:08:13 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120
Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Received higher prio advert
Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Entering BACKUP STATE
Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) removing protocol VIPs.
Dec 12 10:08:15 server12 Keepalived_healthcheckers[3106]: Netlink reflector reports IP 192.168.1.120 removed
Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: Netlink reflector reports IP 192.168.1.120 removed
Dec 12 10:08:15 server12 avahi-daemon[2921]: Withdrawing address record for 192.168.1.120 on eth0.

从实例角色转换日志：
[root@server12 ~]# tail -f /usr/local/redis2/var/keepalived-redis-state.log
[backup]
Wed Dec 12 10:08:15 CST 2012
Being slave….
Run SLAVEOF cmd …
OK