Keepalived 架构和安装-李佳程的个人主页

1、Keepalived 介绍

vrrp 协议的软件实现，原生设计目的为了高可用 ipvs服务

官网：http://keepalived.org/

功能：

基于vrrp协议完成地址流动
为vip地址所在的节点生成ipvs规则(在配置文件中预先定义)
为ipvs集群的各RS做健康状态检测
基于脚本调用接口完成脚本中定义的功能，进而影响集群事务，以此支持nginx、haproxy等服务

2、Keepalived 架构

用户空间核心组件：
- vrrp stack：VIP消息通告
- checkers：监测 Real Server
- system call：实现 vrrp 协议状态转换时调用脚本的功能
- SMTP：邮件组件
- IPVS wrapper：生成 IPVS 规则
- Netlink Reflector：网络接口
- WatchDog：监控进程
控制组件：提供keepalived.conf 的解析器，完成Keepalived配置
IO复用器：针对网络目的而优化的自己的线程抽象
内存管理组件：为某些通用的内存管理功能（例如分配，重新分配，发布等）提供访问权限

# Keepalived进程树

Keepalived <-- Parent process monitoring children
\_ Keepalived <-- VRRP child
\_ Keepalived <-- Healthchecking child

3、环境准备

各节点时间必须同步：ntp, chrony
关闭防火墙及SELinux
各节点之间可通过主机名互相通信：非必须
建议使用/etc/hosts文件实现：非必须
各节点之间的root用户可以基于密钥认证的ssh服务完成互相通信：非必须

4、相关文件

软件包名：keepalived
主程序文件：/usr/sbin/keepalived
主配置文件：/etc/keepalived/keepalived.conf
配置文件示例：/usr/share/doc/keepalived/
Unit File：/lib/systemd/system/keepalived.service
Unit File的环境配置文件：
- /etc/sysconfig/keepalived CentOS
- /etc/default/keepalived Ubuntu

5、Keepalived 安装

# 包安装
#CentOS
[root@centos ~]#yum install keepalived
#ubuntu
[root@ubuntu1804 ~]#apt -y install keepalived

# 编译安装
[root@haproxy01 ~]# yum install -y gcc curl openssl-devel libnl3-devel net-snmp-devel ipvsadm

[root@haproxy01 ~]# wget https://keepalived.org/software/keepalived-2.2.7.tar.gz --no-check-certificate

[root@haproxy01 ~]# tar xvf keepalived-2.2.7.tar.gz -C /usr/local/src

[root@haproxy01 ~]# cd /usr/local/src/keepalived-2.2.7/

# 选项--disable-fwmark 可用于禁用iptables规则,可访止VIP无法访问,无此选项默认会启用iptables
规则
[root@haproxy01 keepalived-2.2.7]# ./configure --prefix=/usr/local/keepalived --disable-fwmark
[root@haproxy01 keepalived-2.2.7]# make && make install


[root@haproxy01 keepalived-2.2.7]# cd /usr/local/keepalived/sbin/

[root@haproxy01 sbin]# ./keepalived -v
Keepalived v2.2.7 (01/16,2022)

Copyright(C) 2001-2022 Alexandre Cassen, <acassen@gmail.com>

Built with kernel headers for Linux 3.10.0
Running on Linux 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020
Distro: CentOS Linux 7 (Core)

configure options: --prefix=/usr/local/keepalived --disable-fwmark

Config options:  LVS VRRP VRRP_AUTH VRRP_VMAC OLD_CHKSUM_COMPAT INIT=systemd SYSTEMD_NOTIFY

System options:  VSYSLOG LIBNL3 RTA_ENCAP RTA_EXPIRES RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTA_VIA IFA_FLAGS NET_LINUX_IF_H_COLLISION LIBIPTC_LINUX_NET_IF_H_COLLISION LIBIPVS_NETLINK IFLA_LINK_NETNSID GLOB_BRACE GLOB_ALTDIRFUNC INET6_ADDR_GEN_MODE

# 默认会自动生成unit文件
[root@haproxy01 keepalived-2.2.7]# cp keepalived/keepalived.service /lib/systemd/system/

[root@haproxy01 keepalived-2.2.7]# cat /lib/systemd/system/keepalived.service
[Unit]
Description=LVS and VRRP High Availability Monitor
After=network-online.target syslog.target
Wants=network-online.target
Documentation=man:keepalived(8)
Documentation=man:keepalived.conf(5)
Documentation=man:genhash(1)
Documentation=https://keepalived.org

[Service]
Type=notify
NotifyAccess=all
PIDFile=/run/keepalived.pid
KillMode=process
EnvironmentFile=-/usr/local/keepalived/etc/sysconfig/keepalived
ExecStart=/usr/local/keepalived/sbin/keepalived --dont-fork $KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

# 创建配置文件
[root@haproxy01 ~]# mkdir /etc/keepalived
[root@haproxy01 ~]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf.sample /etc/keepalived/keepalived.conf

# 启动服务
[root@haproxy01 ~]# systemctl enable --now keepalived.service
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.
[root@haproxy01 ~]# systemctl status keepalived.service
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-12-16 15:43:02 CST; 6s ago
     Docs: man:keepalived(8)
           man:keepalived.conf(5)
           man:genhash(1)
           https://keepalived.org
 Main PID: 13534 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─13534 /usr/local/keepalived/sbin/keepalived --dont-fork -D
           ├─13535 /usr/local/keepalived/sbin/keepalived --dont-fork -D
           └─13536 /usr/local/keepalived/sbin/keepalived --dont-fork -D

Dec 16 15:43:06 haproxy01 Keepalived_vrrp[13536]: Sending gratuitous ARP on eth0 for 192.168.200.18
Dec 16 15:43:07 haproxy01 Keepalived_healthcheckers[13535]: HTTP_CHECK on service [192.168.200.2]:tcp:1358 failed after 3 retries.
Dec 16 15:43:07 haproxy01 Keepalived_healthcheckers[13535]: Removing service [192.168.200.2]:tcp:1358 from VS [10.10.10.2]:tcp:1358
Dec 16 15:43:08 haproxy01 Keepalived_healthcheckers[13535]: HTTP_CHECK on service [192.168.200.4]:tcp:1358 failed after 3 retries.
Dec 16 15:43:08 haproxy01 Keepalived_healthcheckers[13535]: Removing service [192.168.200.4]:tcp:1358 from VS [10.10.10.3]:tcp:1358
Dec 16 15:43:08 haproxy01 Keepalived_healthcheckers[13535]: HTTP_CHECK on service [192.168.200.3]:tcp:1358 failed after 3 retries.
Dec 16 15:43:08 haproxy01 Keepalived_healthcheckers[13535]: Removing service [192.168.200.3]:tcp:1358 from VS [10.10.10.2]:tcp:1358
Dec 16 15:43:08 haproxy01 Keepalived_healthcheckers[13535]: Lost quorum 1-0=1 > 0 for VS [10.10.10.2]:tcp:1358
Dec 16 15:43:08 haproxy01 Keepalived_healthcheckers[13535]: Adding sorry server [192.168.200.200]:tcp:1358 to VS [10.10.10....:1358
Dec 16 15:43:08 haproxy01 Keepalived_healthcheckers[13535]: Removing alive servers from the pool for VS [10.10.10.2]:tcp:1358
Hint: Some lines were ellipsized, use -l to show in full.

[root@haproxy01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:b0:78:30 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.61/24 brd 192.168.1.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 192.168.200.16/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.200.17/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.168.200.18/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:feb0:7830/64 scope link
       valid_lft forever preferred_lft forever

[root@haproxy01 ~]# ipvsadm --list
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.200.100:https rr persistent 50
TCP  10.10.10.2:connlcli rr persistent 50
  -> 192.168.200.200:connlcli     Masq    1      0          0
TCP  10.10.10.3:connlcli rr persistent 50

6、配置说明

6.1、配置文件组成部分

配置文件

/etc/keepalived/keepalived.conf

配置文件组成

GLOBAL CONFIGURATION
- Global definitions：定义邮件配置，route_id，vrrp配置，多播地址等
VRRP CONFIGURATION
- VRRP instance(s)：定义每个vrrp虚拟路由器
LVS CONFIGURATION
- Virtual server group(s)
- Virtual server(s)：LVS集群的VS和RS

6.2、配置语法说明

全局配置

#/etc/keepalived/keepalived.conf
global_defs {
  notification_email {
    root@localhost #keepalived 发生故障切换时邮件发送的目标邮箱，可以按行区分写
多个
    root@lijiach.com
    252409868@qq.com
  }
  notification_email_from keepalived@localhost  #发邮件的地址
  smtp_server 127.0.0.1     #邮件服务器地址
  smtp_connect_timeout 30   #邮件服务器连接timeout
  router_id ka1.example.com #每个keepalived主机唯一标识，建议使用当前主机名，如果多节点重
名可能会影响切换脚本执行
  vrrp_skip_check_adv_addr  #对所有通告报文都检查，会比较消耗性能，启用此配置后，如果收到的
通告报文和上一个报文是同一个路由器，则跳过检查，默认值为全检查
  vrrp_strict #严格遵守VRRP协议,启用此项后以下状况将无法启动服务:1.无VIP地址 2.配置了单播邻
居 3.在VRRP版本2中有IPv6地址，开启动此项并且没有配置vrrp_iptables时会自动开启iptables防火
墙规则，默认导致VIP无法访问,建议不加此项配置
  vrrp_garp_interval 0 #gratuitous ARP messages 报文发送延迟，0表示不延迟
  vrrp_gna_interval 0  #unsolicited NA messages （不请自来）消息发送延迟
  vrrp_mcast_group4 224.0.0.18 #指定组播IP地址范围：224.0.0.0到239.255.255.255,默认
值：224.0.0.18
  vrrp_iptables        #此项和vrrp_strict同时开启时，则不会添加防火墙规则,如果无配置
vrrp_strict项,则无需启用此项配置
}

配置虚拟路由器

vrrp_instance <STRING> { #<String>为vrrp的实例名,一般为业务名称
  配置参数
  ......
}

# 配置参数：
state MASTER|BACKUP#当前节点在此虚拟路由器上的初始状态，状态为MASTER或者BACKUP
interface IFACE_NAME #绑定为当前虚拟路由器使用的物理接口，如：eth0,bond0,br0,可以和VIP不
在一个网卡

virtual_router_id VRID #每个虚拟路由器唯一标识，范围：0-255，每个虚拟路由器此值必须唯一，否则服务无法启动，同属一个虚拟路由器的多个keepalived节点必须相同,务必要确认在同一网络中此值必须唯一

priority 100 #当前物理节点在此虚拟路由器的优先级，范围：1-254，每个keepalived主机节点此值不同
advert_int 1 #vrrp通告的时间间隔，默认1s
authentication { #认证机制
  auth_type AH|PASS   #AH为IPSEC认证(不推荐),PASS为简单密码(建议使用)
  auth_pass <PASSWORD> #预共享密钥，仅前8位有效，同一个虚拟路由器的多个keepalived节点必须一样
}
virtual_ipaddress { #虚拟IP,生产环境可能指定上百个IP地址
  <IPADDR>/<MASK> brd <IPADDR> dev <STRING> scope <SCOPE> label <LABEL>
  192.168.200.100 #指定VIP，不指定网卡，默认为eth0,注意：不指定/prefix,默认
为/32
  192.168.200.101/24 dev eth1   #指定VIP的网卡，建议和interface指令指定的网卡不在一个
网卡
 192.168.200.102/24 dev eth2 label eth2:1 #指定VIP的网卡label
}
track_interface { #配置监控网络接口，一旦出现故障，则转为FAULT状态实现地址转移
  eth0
  eth1
  …
}

范例

! Configuration File for keepalived
global_defs {
    notification_email {
        acassen@firewall.loc
        failover@firewall.loc
        sysadmin@firewall.loc
    }
    notification_email_from Alexandre.Cassen@firewall.loc
    smtp_server 192.168.200.1
    smtp_connect_timeout 30
    router_id LVS_DEVEL
    vrrp_skip_check_adv_addr
    vrrp_strict                     # 开启限制，会自动生效防火墙设置，导致无访问VIP
    vrrp_garp_interval 0
    vrrp_gna_interval 0
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 80            # 修改此行
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.200.16
        192.168.200.17
        192.168.200.18
    }
}

7、启用 Keepalived 日志功能

# vim /etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-D -S 6"

# vim /etc/rsyslog.conf
local6.*                        /var/log/keepalived.log

8、实现 Keepalived 独立子配置文件

当生产环境复杂时， /etc/keepalived/keepalived.conf 文件中内容过多，不易管理，可以将不同集群的配置，比如：不同集群的VIP配置放在独立的子配置文件中
利用include 指令可以实现包含子配置文件

include /path/file

# mkdir /etc/keepalived/conf.d/
# vim /etc/keepalived/keepalived.conf
global_defs {
    notification_email {
         252409868@qq.com
    }
    notification_email_from 29308620@qq.com
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
    router_id ha1
    vrrp_skip_check_adv_addr
    vrrp_garp_interval 0
    vrrp_gna_interval 0
}
include /etc/keepalived/conf.d/*.conf   #将VRRP相关配置放在子配置文件中

# vim /etc/keepalived/conf.d/cluster1.conf