1 Keepalived 架构和安装
Keepalived进程树
Keepalived <-- Parent process monitoring children \_ Keepalived <-- VRRP child \_ Keepalived <-- Healthchecking child
#环境准备 #两台keepalive机器分别配一个单独网卡用于keepalive做心跳监测 : eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> ...inet 192.168.10.151/24 brd 192.168.10.255 scope global eth1#两台后端服务器服务准备 [root@web01 ~]#apt update & apt install nginx mysql-server -y#各节点时间必须同步,ubuntu自带时间同步功能
#CentOS [root@centos ~]#yum -y install keepalived #ubuntu [root@ubuntu1804 ~]#apt update && apt -y install keepalived
#Ubuntu20.04和22.04安装相关包 [root@ka1 ~]#apt update && apt -y install make gcc ipvsadm build-essential pkg-config automake autoconf libipset-dev libnl-3-dev libnl-genl-3-dev libssl-dev libxtables-dev libip4tc-dev libip6tc-dev libmagic-dev libsnmp-dev libglib2.0-dev libpcre2-dev libnftnl-dev libmnl-dev libsystemd-dev#keepalived官网下载 keepalived-2.2.8.tar.gz #下载解压 [root@ka1 ~]#wget https://keepalived.org/software/keepalived-2.2.8.tar.gz [root@ka1 ~]#tar xvf keepalived-2.2.8.tar.gz -C /usr/local/src [root@ka1 ~]#cd /usr/local/src/keepalived-2.2.8/ #编译 [root@ka1 keepalived-2.2.8]#./configure --prefix=/usr/local/keepalived [root@ka1 keepalived-2.2.8]#make && make install #拷贝已经生成的service文件 [root@ka1 keepalived-2.2.8]#cp keepalived/keepalived.service /lib/systemd/system/ [root@ka1 keepalived-2.2.8]#systemctl daemon-reload#创建配置文件，默认配置路径可以是编译路径/usr/local/keepalived/etc/keepalived/keepalived.conf（优先高）或者/etc/keepalived/keepalived.conf（优先级低） [root@ka1 keepalived-2.2.8]#mkdir /etc/keepalived [root@ka1 keepalived-2.2.8]#cp /usr/local/keepalived/etc/keepalived/keepalived.conf.sample /etc/keepalived/keepalived.conf [root@ka1 keepalived-2.2.8]#systemctl enable --now keepalived.service#ping不通自己的vip，因新版内核嵌了新防火墙nft(centos8开始有了,ubuntu也有) #新版：nftables规则替代iptables规则 [root@ubuntu2204 ~]#nft list ruleset table ip keepalived {set vips {type ipv4_addrelements = { 192.168.200.16, 192.168.200.17,192.168.200.18 }}chain out {type filter hook output priority filter - 1; policy accept;ip saddr @vips drop}chain in {type filter hook input priority filter - 1; policy accept;ip daddr @vips drop} }#清空nft规则 [root@ubuntu2204 ~]#nft flush ruleset [root@ubuntu2204 ~]#nft list ruleset #可以ping通自己的vip
Docker的官方仓库上没有提供Keepalived的容器镜像，可以利用在源码目录中提供的Dockerfile自行制作
全局配置
#/etc/keepalived/keepalived.conf global_defs {notification_email {root@localhost #keepalived 发生故障切换时邮件发送的目标邮箱，可以按行区分写多个 root@wangxiaochun.com 29308620@qq.com }notification_email_from keepalived@localhost #发邮件的地址smtp_server 127.0.0.1 #邮件服务器地址smtp_connect_timeout 30 #邮件服务器连接timeoutrouter_id ka1.example.com #每个keepalived主机唯一标识，建议使用当前主机名，如果多节点重名可能会影响切换脚本执行vrrp_skip_check_adv_addr #默认会对所有通告报文都检查，会比较消耗性能，启用此配置后，如果收到的通告报文和上一个报文是同一个路由器，则跳过检查vrrp_strict #严格遵守VRRP协议,启用此项后以下状况将无法启动服务或工作异常:1.无VIP地址 2.配置了单播邻居 3.在VRRP版本2中有IPv6地址，开启动此项并且没有配置vrrp_iptables时会自动开启iptables(旧内核)或者nft(新内核)的防火墙规则，默认导致VIP无法访问,建议不加此项配置vrrp_garp_interval 0 #gratuitous ARP messages 报文发送延迟，0表示不延迟vrrp_gna_interval 0 #unsolicited NA messages （不请自来）消息发送延迟vrrp_mcast_group4 224.0.0.18 #此为默认多播地址,可以指定。指定组播IP地址范围：224.0.0.0到239.255.255.255,默认值：224.0.0.18，如果配置了单播，此项失效vrrp_iptables #此项和vrrp_strict同时开启时，则不会添加防火墙规则,如果无配置vrrp_strict项,则无需启用此项配置，注意：新版加此项仍有iptables(旧内核)或者nft(新内核)规则 }
精简配置
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf global_defs {router_id ka1 }vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 51priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.200.16192.168.200.17192.168.200.18} } [root@ka1 ~]#systemctl reload keepalived.service
配置虚拟路由器
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf global_defs {router_id ka1 }vrrp_instance VI_1 {state MASTERinterface eth1virtual_router_id 66priority 100advert_int 1authentication {auth_type PASSauth_pass 123456}virtual_ipaddress { #子网掩码默认32,别人会ping不通,这里改2410.0.0.10/24 dev eth0 label eth0:1 #绑定网卡(业务网卡),定个标签别名 } } [root@ka1 ~]#systemctl reload keepalived.service #查看 [root@ka1 ~]#ip a#ka2服务器安装keepalived，修改配置 [root@ka2 ~]#vim /etc/keepalived/keepalived.conf global_defs {router_id ka2 }vrrp_instance VI_1 {state BACKUPinterface eth1virtual_router_id 66priority 80advert_int 1authentication {auth_type PASSauth_pass 123456}virtual_ipaddress { 10.0.0.10/24 dev eth0 label eth0:1} } [root@ka1 ~]#systemctl reload keepalived.service
当生产环境复杂时， /etc/keepalived/keepalived.conf 文件中保存所有集群的配置会导致内容过多，不易管理
利用include 指令可以实现包含子配置文件
include /path/file
范例
[root@ka1 ~]#vim /etc/keepalived/keepalived.conf global_defs {router_id ka1 }include /etc/keepalived/conf.d/*.conf #将VRRP相关配置放在子配置文件中 [root@ka1 ~]#mkdir /etc/keepalived/conf.d [root@ka1 ~]#vim /etc/keepalived/conf.d/www.magedu.org.conf vrrp_instance VI_1 {state MASTERinterface eth1virtual_router_id 66priority 100advert_int 1authentication {auth_type PASSauth_pass 123456}virtual_ipaddress {10.0.0.10/24 dev eth0 label eth0:1} } [root@ka1 ~]#systemctl reload keepalived.service #可能有问题，重启下 [root@ka2 keepalived-2.2.8]#systemctl restart keepalived.service
查看是否脑裂
#通过arping命令，如果返回多条，说明脑裂，多个mac地址有该ip -c表示测几次 [root@ka2 ~]#arping -c1 -I eth0 10.0.0.10
#根据keepalived的service文件，在启动的环境变量里配置或者直接在启动命令里加选项 [root@ka1 ~]#vim /usr/local/keepalived/etc/sysconfig/keepalived KEEPALIVED_OPTIONS="-D -S 6" #0-7选个没人用的,这里写local6#去查看local6有没有定义 [root@ka1 ~]#vim /etc/rsyslog.conf #上面没有，到上面的子配置中查找 [root@ka1 ~]#cd /etc/rsyslog.d [root@ka1 rsyslog.d]#vim 50-default.conf #追加 local6.* /var/log/keepalived.log#重启服务 [root@ka1 rsyslog.d]#systemctl restart keepalived.service rsyslog.service#查看(没有就重启) [root@ka1 rsyslog.d]#tail /var/log/keepalived.log
3 Keepalived 实现 VRRP
3.1 实现Master/Backup的 Keepalived 单主架构
上面已经展示过了
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.magedu.org.conf #ha1主机配置 vrrp_instance VI_1 {state BACKUP #都为BACKUP interface eth1virtual_router_id 66priority 100 #优先级高advert_int 1nopreempt #添加此行，设为nopreempt#ha2主机配置 vrrp_instance VI_1 {state BACKUP #都为BACKUP interface eth1virtual_router_id 66priority 80 #优先级低advert_int 1#nopreempt #注意：如果ka2主机也是非抢占式，会导致ka1即使优先级降低于ka2，VIP也不会切换至ka2 [root@ka1 ~]#systemctl reload keepalived.service
但是如果低优先级的主机down机，则立即抢占VIP地址，而不再延迟
preempt_delay # #指定抢占延迟时间为#s，默认延迟300s，在优先级高的节点配置
范例：
#ka1主机配置 vrrp_instance VI_1 {state BACKUP #都为BACKUP interface eth0virtual_router_id 66priority 100 #优先级高advert_int 1preempt_delay 60 #抢占延迟模式，默认延迟300s#ka2主机配置 vrrp_instance VI_1 {state BACKUP #都为BACKUP interface eth0virtual_router_id 66priority 80 #优先级低advert_int 1
默认keepalived主机之间利用多播相互通告消息，会造成网络拥塞，可以设置为单播，减少网络流量
单播优先与多播，即同时配置，单播生效
#在所有节点vrrp_instance语句块中设置对方主机的IP，建议设置为专用于对应心跳线网络的地址，而非使 用业务网络 unicast_src_ip <IPADDR> #指定发送单播的源IP unicast_peer {<IPADDR> #指定接收单播的对方目标主机IP(多个就写多个) ...... }
范例：
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.magedu.org.conf vrrp_instance VI_1 {state MASTERinterface eth1virtual_router_id 66priority 100advert_int 1unicast_src_ip 192.168.10.151unicast_peer {192.168.10.152}authentication {auth_type PASSauth_pass 123456}virtual_ipaddress {10.0.0.10/24 dev eth0 label eth0:1} } [root@ka1 ~]#systemctl restart keepalived.service [root@ka2 ~]#vim /etc/keepalived/conf.d/www.magedu.org.conf vrrp_instance VI_1 {state BACKUPinterface eth1virtual_router_id 66priority 80advert_int 1unicast_src_ip 192.168.10.152unicast_peer {192.168.10.151}authentication {auth_type PASSauth_pass 123456}virtual_ipaddress {10.0.0.10/24 dev eth0 label eth0:1} } [root@ka2 ~]#systemctl restart keepalived.service
当keepalived的状态变化时，可以自动触发脚本的执行，比如：发邮件通知用户
默认以用户keepalived_script身份执行脚本，如果此用户不存在，以root执行脚本
可以用下面指令指定脚本执行用户的身份
global_defs {......script_user <USER>...... }
notify_master <STRING>|<QUOTED-STRING>
当前节点转为备节点时触发的脚本
notify_backup <STRING>|<QUOTED-STRING>
当前节点转为“失败”状态时触发的脚本
notify_fault <STRING>|<QUOTED-STRING>
通用格式的通知触发机制，一个脚本可完成以上三种状态的转换时的通知
notify <STRING>|<QUOTED-STRING>
当停止VRRP时触发的脚本
notify_stop <STRING>|<QUOTED-STRING>
#记得给脚本添加执行权限 notify_master "/etc/keepalived/notify.sh master" #执行的命令可以自己更改 notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault"
范例
#两台keepalive机器结尾都追加,发送邮箱脚本 [root@ka1 ~]#vim /etc/keepalived/conf.d/www.magedu.org.conf vrrp_instance VI_1 {state MASTERinterface eth1virtual_router_id 66priority 100advert_int 1unicast_src_ip 192.168.10.151unicast_peer {192.168.10.152}authentication {auth_type PASSauth_pass 123456}virtual_ipaddress {10.0.0.10/24 dev eth0 label eth0:1}notify_master "/etc/keepalived/notify.sh master" #追加notify_backup "/etc/keepalived/notify.sh backup"notify_fault "/etc/keepalived/notify.sh fault" } [root@ka1 ~]#systemctl reload keepalived.service
访问量大，不要用这种模式
#复制一份新的子配置文件 [root@ka1 keepalived]#vim /etc/keepalived/conf.d/www.magedu.net.conf vrrp_instance VI_2 { #修改名称state BACKUP #修改角色 interface eth1virtual_router_id 88 #修改priority 80 #修改优先级advert_int 1unicast_src_ip 192.168.10.151 #可共用vrrp单播,同一个心跳网络 unicast_peer {192.168.10.152}authentication {auth_type PASSauth_pass 654321}virtual_ipaddress {10.0.0.20/24 dev eth0 label eth0:2 #修改vip,修改别名 }notify_master "/etc/keepalived/notify.sh master"notify_backup "/etc/keepalived/notify.sh backup"notify_fault "/etc/keepalived/notify.sh fault" } [root@ka1 ~]#systemctl restart keepalived.service [root@ka2 keepalived]#vim /etc/keepalived/conf.d/www.magedu.net.conf vrrp_instance VI_2 { #修改名称state MASTER #修改角色 interface eth1virtual_router_id 88 #修改priority 100 #修改优先级advert_int 1unicast_src_ip 192.168.10.151 #可共用vrrp单播,同一个心跳网络 unicast_peer {192.168.10.152}authentication {auth_type PASSauth_pass 654321}virtual_ipaddress {10.0.0.20/24 dev eth0 label eth0:2 #修改vip,修改别名 }notify_master "/etc/keepalived/notify.sh master"notify_backup "/etc/keepalived/notify.sh backup"notify_fault "/etc/keepalived/notify.sh fault" } [root@ka2 ~]#systemctl restart keepalived.service
不放心，可以多配几个从节点。比如8台机器，可以配成一主两从
vrrp_sync_group VG_1 {group { #作为一组一起漂移VI_1 # name of vrrp_instance (below)VI_2 # One for each moveable IP }} vrrp_instance VI_1 {eth0vip} vrrp_instance VI_2 {eth1dip}
解决lvs两大问题 1.高可用问题 2.监测后端服务是否正常，如果异常就剔除该服务，恢复再加入 #443属于https,加密七层。一般使用四层检查
lvs内核监控，外部无法看到其对应监控的端口
#查看lvs规则 ipvsadm -Ln
vrrp_script <SCRIPT_NAME> { #定义一个检测脚本，在global_defs 之外配置script <STRING>|<QUOTED-STRING> #shell命令或脚本路径interval <INTEGER> #间隔时间，单位为秒，默认1秒timeout <INTEGER> #超时时间weight <INTEGER:-254..254> #默认为0,如果设置此值为负数，当上面脚本返回值为非0时，会将此值与本节点权重相加可以降低本节点权重，即表示fall. 如果是正数，当脚本返回值为0，会将此值与本节点权重相加可以提高本节点权重，即表示 rise.通常使用负值fall <INTEGER> #执行脚本连续几次都失败,则转换为失败，建议设为2以上rise <INTEGER> #执行脚本连续几次都成功，把服务器从失败标记为成功user USERNAME [GROUPNAME] #执行监测脚本的用户或组 init_fail #设置默认标记为失败状态，监测成功之后再转换为成功状态
}
vrrp_instance VI_1 {…track_script {<SCRIPT_NAME>} }
[root@ka1 ~]#tcpdump -i eth1 -nn hsot 192.168.10.151
[root@ka1 ~]#cat /etc/keepalived/keepalived.conf global_defs {... }vrrp_script check_down {#存在文件为假就降优先级为70,当不存在该文件又恢复优先级为100script "[ ! -f /etc/keepalived/down ]" #/etc/keepalived/down存在时返回非0，触发权重-30interval 1weight -30 #优先级prio-30,从而出发地址漂移fall 3rise 2timeout 2 }vrrp_instance VI_1 {state MASTER #在另一个节点为BACKUP interface eth0virtual_router_id 66priority 100 #在另一个节点为80advert_int 1authentication {auth_type PASSauth_pass 123456}virtual_ipaddress {10.0.0.10/24 dev eth0 label eth0:1}track_interface {eth0}notify_master "/etc/keepalived/notify.sh master"notify_backup "/etc/keepalived/notify.sh backup"notify_fault "/etc/keepalived/notify.sh fault"track_script {check_down #调用前面定义的脚本 } }
#两台keepalived服务器安装haproxy [root@ka1 ~]#apt install haproxy [root@ka1 ~]#vim /etc/haproxy/haproxy.cfg listen status #追加状态页 mode httpbind 0.0.0.0:9999stats enablelog globalstats uri /haproxy-statuslisten webserversbind 10.0.0.10:80server web01 10.0.0.153:80 checkserver web02 10.0.0.154:80 check [root@ka1 ~]#systemctl restart haproxy.service#在另一个台配置haproxy.cfg时，vip10.0.0.10不在上面，导致绑定不上去报错，要改内核参数(两台都要改) #查看内核参数 [root@ka2 ~]#sysctl -a|grep bind net.ipv4.ip_nonlocal_bind = 0 [root@ka2 ~]#vim /etc/sysctl.conf #追加 net.ipv4.ip_nonlocal_bind = 1 #允许绑定不存在的ip端口，允许使用 #从配置文件“/etc/sysctl.conf”加载内核参数设置 [root@ka2 ~]#sysctl -p net.ipv4.ip_nonlocal_bind = 1 #再修改/etc/haproxy/haproxy.cfg即可绑定#在第一个keepalived节点配，第二个节点不用配(减了也没人替) [root@ka1 ~]#vim /etc/keepalived/conf.d/www.magedu.org.conf vrrp_script chk_haproxy { #严谨判断写脚本判断状态页script "killall -0 haproxy" #检查haproxy(只能监测进程没事,不能说明进程健康)interval 1weight -30fall 2rise 2 } vrrp_instance VI_1 {state MASTERinterface eth1virtual_router_id 66priority 100advert_int 1unicast_src_ip 192.168.10.151unicast_peer {192.168.10.152}authentication {auth_type PASSauth_pass 123456}virtual_ipaddress {10.0.0.10/24 dev eth0 label eth0:1}track_script {chk_haproxy #调用前面定义的脚本 } } [root@ka1 ~]#systemctl restart keepalived.service#测试关闭ka1上的keepalived,看nginx端请求是否来自ka2; #启动ka1上的keepalived,看nginx端请求是否来自ka1(ka1又抢回vip)
nginx同上(内核参数可以不改)，把检查haproxy换成检查nginx killall -0 nginx
#探测127.0.0.1的9999端口是否打开(当然端口打开,进程也有可能假死,也不算特别严谨) [root@ka1 ~]#</dev/tcp/127.0.0.1/9999 #读取数据到当前终端 [root@ka1 ~]#echo $? 0 [root@ka1 ~]#</dev/tcp/127.0.0.1/9998 -bash: connect: Connection refused -bash: /dev/tcp/127.0.0.1/9998: Connection refused [root@ka1 ~]#echo $? 1
检查脚本可以改成尝试重启，如果失败再降低优先级进行切换
[root@ka1 ~]#vim /etc/keepalived/conf.d/www.magedu.org.conf vrrp_script chk_nginx {script "/etc/keepalived/check_nginx.sh"interval 1weight -30fall 2rise 2 }[root@ka1 ~]#vim /etc/keepalived/check_nginx.sh #如检查nginx成功,返回成功;失败重启nginx,成功返回成功,失败返回失败 killall -0 nginx || systemctl restart nginx
