⼀、keepalived⾼可⽤简介

keepalived是⼀个类似与layer3、4和7交换机制的软件，keepalived软件有两种功能，分别是监控检查、VRRP(虚拟路由器冗余协议)

keepalived的作⽤是检测Web服务器的状态，⽐如有⼀台Web服务器、MySQL服务器宕机或⼯作出现故障，keepalived检测到后，会将故障的Web服务器或者MySQL服务器从系统中剔除，当服务器⼯作正常后keepalived⾃动将服务器加⼊到服务器群中，这些⼯作全部⾃动完成，不需要⼈⼯⼲涉，需要⼈⼯做的值是修复故障的Web和MySQL服务器。layer3、4、7⼯作在TCP/IP协议栈的IP层、传输层、应⽤层，实现原理为：

layer3：keepalived使⽤layer3的⽅式⼯作时，keepalived会定期向服务器群中的服务器发送⼀个ICMP数据包，如果发现某台服务的IP地址⽆法ping通，keepalived便报告这台服务器失效，并将它从服务器集群中剔除。layer3的⽅式是以服务器的IP地址是否有效作为服务器⼯作是否正常的标准

layer4：layer4主要以TCP端⼝的状态来决定服务器⼯作是否正常。例如Web服务端⼝⼀般为80，如果keepalived检测到80端⼝没有启动，则keepalived把这台服务器从服务器集群中剔除

layer7：layer7⼯作在应⽤层，keepalived将根据⽤户的设定检查服务器的运⾏是否正常，如果与⽤户的设定不相符，则keepalived将把服务器从服务器集群中剔除⼆、nginx+keepalived集群1、原理及环境

Nginx负载均衡⼀般位于整个架构的最前端或者中间层，如果为最前端时单台nginx会存在单点故障，⼀台nginx宕机，会影响⽤户对整个⽹站的访问。如果需要加⼊nginx备份服务器，nginx主服务器与备份服务器之间形成⾼可⽤，⼀旦发现nginx主宕机，能够快速将⽹站切换⾄备份服务器。原理图:

准备环境：

nginx-1：172.25.70.1（master），主机名为：keep1nginx-2：172.25.70.2（backup），主机名为：keep22、安装配置

(1)master和backup均安装nginx

##之前预编译需要的gcc、gcc-c++、openssl、openssl-devel等默认已经安装好[root@keep1 ~]# yum install -y pcre-devel ##安装perl兼容的正则表达式库[root@keep1 ~]# cd nginx-1.12.0

[root@keep1 nginx-1.12.0]# sed -i -e 's/1.12.0//g' -e 's/nginx\\//TDTWS/g' -e 's/\"NGINX\"/\"TDTWS\"/g' src/core/nginx.h ##sed修改Nginx版本信息为TDTWS[root@keep1 nginx-1.12.0]# ./configure --prefix=/usr/local/nginx --user=www --group=www --with-http_stub_status_module --with-http_ssl_module[root@keep1 nginx-1.12.0]# make && make install[root@keep1 ~]# vim /usr/local/nginx/conf/nginx.conf将该⽂件⾥⾯的user nobody的注释去掉

[root@keep1 ~]# ln -s /usr/local/nginx/sbin/nginx /sbin/nginx #创建命令快捷启动[root@keep1 ~]# nginx #没有报错表⽰启动成功

(2)master和backup均安装keepalived

##安装依赖包

[root@keep1 ~]# yum -y install libnl libnl-devel libnfnetlink

此时还需要⼀个包libnfnetlink-devel，但因为redhat6.5⾃⾝的镜像源中没有，所以给⼤家提供⼀个地址，下载了之后直接⽤rpm -ivh安装即可[root@localhost ~]# wget ftp://mirror.switch.ch/mirror/centos/6/os/x86_64/Packages/libnfnetlink-devel-1.0.0-1.el6.x86_64.rpm[root@keep1 keepalived-1.4.3]# rpm -ivh libnfnetlink-devel-1.0.0-1.el6.x86_64.rpm##编译安装

[root@keep1 ~]# tar zxf keepalived-1.3.6.tar.gz [root@keep1 ~]# cd keepalived-1.3.6

[root@keep1 keepalived-1.3.6]# ./configure --prefix=/usr/local/keepalived --with-init=SYSV[root@keep1 keepalived-1.3.6]# make && make install

##做启动链接等

[root@keep1 keepalived-1.3.6]# ln -s /usr/local/keepalived/etc/keepalived /etc/

[root@keep1 keepalived-1.3.6]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/[root@keep1 keepalived-1.3.6]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/[root@keep1 keepalived-1.3.6]# ln -s /usr/local/keepalived/sbin/keepalived /sbin/

[root@keep1 keepalived-1.3.6]# chmod +x /usr/local/keepalived/etc/rc.d/init.d/keepalived

(3)master和backup分别配置keepalived配置⽂件master

[root@keep1 ~]# vim /etc/keepalived/keepalived.confglobal_defs {

notification_email {

root@localhost #健康检查报告通知邮箱}

notification_email_from keepalived@localhost #发送邮件的地址smtp_server 127.0.0.1 #邮件服务器smtp_connect_timeout 30route_id LVS_DEVEL}

vrrp_script_chk_nginx {

script \"/data/sh/check_nginx.sh\" ##检查本地nginx是否存活脚本需要⾃⼰写，后⾯会有该脚本内容interval 2weight 2}

#VIP1

vrrp_instance VI_1 {state BACKUPinterface eth0

lvs_sync_daemon_interface eth0virtual_router_id 151priority 100

advert_int 5 #健康检测频率nopreempt

authentication {auth_type PASSauth_pass 1111}

virtual_ipaddress {172.25.70.100 ##VIP}

track_script {chk_nginx}}

##以下脚本⽤于检查本地nginx是否存活，如果不存活，则服务实现切换[root@keep1 ~]# vim /data/sh/check_nginx.sh#!/bin/bashkillall -0 nginx

if [[ $? -ne 0 ]]; then

/etc/init.d/keepalived stopfi

##编写⼀个nginx显⽰的html⽂件

[root@keep1 ~]# vim /usr/local/nginx/html/index.html

172.25.70.1

backup

##backup的keepalived的配置⽂件和master只有优先级不⼀样[root@keep2 ~]# vim /etc/keepalived/keepalived.confglobal_defs {

notification_email {

root@localhost #健康检查报告通知邮箱}

notification_email_from keepalived@localhost #发送邮件的地址smtp_server 127.0.0.1 #邮件服务器smtp_connect_timeout 30route_id LVS_DEVEL}

vrrp_script_chk_nginx {

script \"/data/sh/check_nginx.sh\" ##检查本地nginx是否存活脚本需要⾃⼰写，后⾯会有该脚本内容interval 2weight 2}

#VIP1

vrrp_instance VI_1 {state BACKUPinterface eth0

lvs_sync_daemon_interface eth0virtual_router_id 151priority 90

advert_int 5 #健康检测频率nopreempt

authentication {auth_type PASSauth_pass 1111}

virtual_ipaddress {172.25.70.100 ##VIP}

track_script {chk_nginx}}

##backup和master的/data/sh/check_nginx.sh⽂件相同，这⾥就不再显⽰了##编写⼀个nginx显⽰的html⽂件

[root@keep2 ~]# vim /usr/local/nginx/html/index.html

172.25.70.2

3、测试

1、两台主机的nginx和keepalived都正常⼯作，使⽤浏览器访问虚拟ip 172.25.70.100应该得到keep1主机的nginx页⾯

2、关闭keep1的keepalived，再⽤浏览器访问虚拟ip查看是否实现了⾼可⽤

如果在真实情况中，主的nginx宕掉了，两个nginx页⾯⼀致，那么会快速将⽹站切换到备份的服务器上⾯去

得到上图结果，表⽰该实验成功！

三、nginx+keepalived双主架构

nginx+keepalived主备模式，始终有⼀台服务器处于空闲状态。为了更好地利⽤服务器，可以把它们设置为双主模式，另⼀台为这⼀台的备份，同时它⼜是另外⼀个VIP的主服务器，两台同时对外提供不同服务，同时接收⽤户的请求。原理图： 环境：

keep1:172.25.70.1keep2：172.25.70.2

VIP1：172.25.70.100 ，主为keep1，从为keep2VIP2：172.25.70.150，主为keep2，从为keep12、配置⽂件

(1)kepp1主机配置keepalived.conf

其实跟上⾯的集群都是⼀个套路，所以这⾥就没有注释了keep1主机keepalived.conf配置⽂件内容如下：

[root@keep1 ~]# vim /etc/keepalived/keepalived.confglobal_defs {

notification_email {root@localhost}

notification_email_from keepalived@localhostsmtp_server 127.0.0.1smtp_connect_timeout 30route_id LVS_DEVEL}

vrrp_script_chk_nginx {

script \"/data/sh/check_nginx.sh\"interval 2weight 2}

#VIP1

vrrp_instance VI_1 {state MASTER

interface eth0

lvs_sync_daemon_interface eth0virtual_router_id 151priority 100advert_int 5nopreempt

authentication {auth_type PASSauth_pass 1111}

virtual_ipaddress {172.25.70.100}

track_script {chk_nginx}}

#VIP2

vrrp_instance VI_2 {state BACKUPinterface eth0

lvs_sync_daemon_interface eth0virtual_router_id 152priority 90advert_int 5nopreempt

authentication {auth_type PASSauth_pass 2222}

virtual_ipaddress {172.25.70.150}

track_script {chk_nginx}}

(2)keep2主机配置keepalived.conf

keep2主机配置keepalived.conf⽂件内容如下：

[root@keep2 ~]# vim /etc/keepalived/keepalived.confglobal_defs {

notification_email {root@localhost}

notification_email_from keepalived@localhostsmtp_server 127.0.0.1smtp_connect_timeout 30route_id LVS_DEVEL}

vrrp_script_chk_nginx {

script \"/data/sh/check_nginx.sh\"interval 2weight 2}

#VIP1

vrrp_instance VI_1 {state BACKUPinterface eth0

lvs_sync_daemon_interface eth0virtual_router_id 151priority 90advert_int 5nopreempt

authentication {auth_type PASSauth_pass 1111}

virtual_ipaddress {172.25.70.100}

track_script {chk_nginx}}

#VIP2

vrrp_instance VI_2 {state MASTERinterface eth0

lvs_sync_daemon_interface eth0virtual_router_id 152priority 100advert_int 5nopreempt

authentication {auth_type PASSauth_pass 2222}

virtual_ipaddress {172.25.70.150}

track_script {chk_nginx}}

配置完成后重新启动服务

(3)两台服务器上检测脚本还是和集群实验中的脚本内容相同3、测试

1、正常情况下，两个虚拟⽹卡在它⾃⼰为主的那个主机上，如下图

2、当其中⼀台主服务器DOWN掉，则会发现宕掉的那个VIP的从机开始⼯作，那么两个VIP此时都会在同⼀个主机上

那么双主架构实验则成功！

4、管理与维护

nginx+keepalived双主架构，⽇常维护和管理需要从以下⼏个⽅⾯：nginx+keepalived⾼可⽤

负载均衡技术对于⼀个⽹站尤其是⼤型⽹站的web服务器集群来说是⾄关重要的！做好负载均衡架构，可以实现故障转移和⾼可⽤环境，避免单点故障，保证⽹站健康持续运⾏。

由于业务扩展，⽹站的访问量不断加⼤，负载越来越⾼。现需要在web前端放置nginx负载均衡,同时结合keepalived对前端nginx实现HA⾼可⽤。

介绍下Nginx和keepalive1.Nginx

Nginx 是⼀个很强⼤的⾼性能Web和反向代理服务器，它具有很多⾮常优越的特性：

Nginx作为负载均衡服务器：Nginx 既可以在内部直接⽀持 Rails 和 PHP 程序对外进⾏服务，也可以⽀持作为 HTTP代理服务器对外进⾏服务。Nginx采⽤C进⾏编写，不论是系统资源开销还是CPU使⽤效率都⽐ Perlbal 要好很多。2.keepalive

Keepalived是Linux下⾯实现VRRP备份路由的⾼可靠性运⾏件。基于Keepalived设计的服务模式能够真正做到主服务器和备份服务器故障时IP瞬间⽆缝交接。⼆者结合，可以构架出⽐较稳定的软件LB⽅案。Nginx+keepalive⾼可⽤⽅式有两种：1.Nginx+keepalived 主从配置

这种⽅案，使⽤⼀个vip地址，前端使⽤2台机器，⼀台做主，⼀台做备，但同时只有⼀台机器⼯作，另⼀台备份机器在主机器不出现故障的时候，永远处于浪费状态，对于服务器不多的⽹站，该⽅案不经济实惠。2.Nginx+keepalived 双主配置

这种⽅案，使⽤两个vip地址，前端使⽤2台机器，互为主备，同时有两台机器⼯作，当其中⼀台机器出现故障，两台机器的请求转移到⼀台机器负担，⾮常适合于当前架构环境。所以在这⾥就详细介绍下双主模型配置

⼀、拓扑结构

⼆、测试环境介绍

系统centos7.4 64位centos6.9 64位

前端node1服务器：DIP:192.168.92.136VIP1:192.168.92.23VIP2：192.168.92.24

前端node2服务器：DIP:192.168.92.133VIP1:192.168.92.24VIP2：192.168.92.23

后端服务器：web node3:192.168.92.123web node4:192.168.92.124web node5:192.168.92.125

我们开始之前先把防⽕墙和selinux关掉，很多时候我们服务器之间不通都是这些原因造成的。

三、软件安装

Nginx和keepalive的安装⾮常简单，我们可以直接使⽤yun来安装。yum install keepalived nginx -y

后端服务器我们同样⽤yum来装上Nginx后端node3

[root@node3 ~]# yum -y install nginx

[root@node3 ~]# echo \"this is 192.168.92.123\" > /usr/share/nginx/html/index.html [root@node3 ~]# service nginx start[root@node3 ~]# curl 192.168.92.123this is 192.168.92.123

后端node4

[root@node4 ~]# yum -y install nginx

[root@node4 ~]# echo \"this is 192.168.92.124\" > /usr/share/nginx/html/index.html [root@node4 ~]# service nginx start[root@node4 ~]# curl 192.168.92.124this is 192.168.92.124

后端node5

[root@node5 ~]# yum -y install nginx

[root@node5 ~]# echo \"this is 192.168.92.125\" > /usr/share/nginx/html/index.html [root@node5 ~]# service nginx start[root@node5 ~]# curl 192.168.92.125this is 192.168.92.125

四、在node1、node2上配置Nginx

[root@node2 ~]# vim /etc/nginx/conf.d/node2.conf #在扩展配置⽬录中配置需要注释掉主配置⽂件中的server部分upstream web1 {

#ip_hash; #hash绑定ip server 192.168.92.123:80; server 192.168.92.124:80; server 192.168.92.125:80; }

server {

listen 80;

server_name www.node.com; index index.html index.htm; location / {

proxy_set_header Host $host;

proxy_set_header X-Forwarded-For $remote_addr; proxy_pass http://web1; } }

五、在node1上配置keepalive

[root@node1 ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalivedglobal_defs {

notification_email { root@localhost

}

notification_email_from Alexandre.Cassen@firewall.loc smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node1

vrrp_mcast_gruop4 224.0.100.23}

vrrp_script chk_haproxy {

script \"/etc/keepalived/chk_nginx.sh\" interval 2 weight 2}

vrrp_instance VI_1 { state MASTER interface ens37 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 111123 }

track_script { chk_nginx }

virtual_ipaddress { 192.168.92.23 }}

vrrp_instance VI_2 { state BACKUP interface ens37

virtual_router_id 151 priority 98 advert_int 1 authentication { auth_type PASS auth_pass 123123 }

track_script { chk_nginx }

virtual_ipaddress { 192.168.92.24 }}

六、在node2上配置keepalive

[root@node2 ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived

global_defs {

notification_email { root@localhost }

notification_email_from Alexandre.Cassen@firewall.loc smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node1

vrrp_mcast_gruop4 224.0.100.23}

vrrp_script chk_haproxy {

script \"/etc/keepalived/chk_nginx.sh\" interval 2 weight 2}

vrrp_instance VI_1 { state BACKUP interface ens34 virtual_router_id 51 priority 98 advert_int 1 authentication { auth_type PASS auth_pass 111123 }

track_script { chk_nginx }

virtual_ipaddress { 192.168.92.23

}}

vrrp_instance VI_2 { state MASTER interface ens34

virtual_router_id 151 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 123123 }

track_script { chk_nginx }

virtual_ipaddress { 192.168.92.24 }}

七、在双主服务器上添加检测脚本

此脚本作⽤是检测Nginx是否运⾏，如果没有运⾏就启动Nginx如果启动失败则停⽌keepalive，保证备⽤服务器正常运⾏。

[root@node2 ~]# cat /etc/keepalived/chk_nginx.sh #!/bin/bash

status=$(ps -C nginx --no-heading|wc -l)if [ \"${status}\" = \"0\" ]; then systemctl start nginx

status2=$(ps -C nginx --no-heading|wc -l) if [ \"${status2}\" = \"0\" ]; then systemctl stop keepalived fifi

⼋、启动Nginx、keepalive服务

[root@node2 ~]# service nginx start

[root@node2 ~]# service keepalived start[root@node3 ~]# service nginx start

[root@node3 ~]# service keepalived start

九、查看VIP并测试访问

[root@node2 ~]# ip a..........

ens34: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:ca:0b:2b brd ff:ff:ff:ff:ff:ff

inet 192.168.92.133/24 brd 192.168.92.255 scope global dynamic ens34 valid_lft 1293sec preferred_lft 1293sec inet 192.168.92.24/32 scope global ens34 valid_lft forever preferred_lft forever

inet6 fe80::9bff:2e2b:aebb:e35/64 scope link valid_lft forever preferred_lft forever.........

[root@node1 ~]# ip a..........

ens37: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:04:b6:17 brd ff:ff:ff:ff:ff:ff

inet 192.168.92.136/24 brd 192.168.92.255 scope global dynamic ens37 valid_lft 1567sec preferred_lft 1567sec inet 192.168.92.23/32 scope global ens37 valid_lft forever preferred_lft forever

inet6 fe80::7ff4:9608:5903:1a4b/64 scope link valid_lft forever preferred_lft forever..........

[root@node1 ~]# curl http://192.168.92.23this is 192.168.92.123

[root@node1 ~]# curl http://192.168.92.23this is 192.168.92.124

[root@node1 ~]# curl http://192.168.92.23this is 192.168.92.125

[root@node1 ~]# curl http://192.168.92.24this is 192.168.92.124

⼗、测试脚本是否能正常运⾏

⼿动停⽌Nginx后⾃动恢复启动

[root@node1 ~]# systemctl stop nginx[root@node1 ~]# ss -tnlp

State Recv-Q Send-Q Local Address:Port Peer Address:Port

LISTEN 0 128 *:80 *:* users:((\"nginx\LISTEN 0 128 *:22 *:* users:((\"sshd\

LISTEN 0 100 127.0.0.1:25 *:* users:((\"master\LISTEN 0 128 :::22 :::* users:((\"sshd\LISTEN 0 100 ::1:25 :::*