LVS+Keepalived+Nginx实现高性能负载均衡集群完整指南

LVS+Keepalived+Nginx实现高性能负载均衡集群完整指南

一、系统架构概述

LVS（Linux Virtual Server）是Linux内核级的负载均衡解决方案，结合Keepalived实现高可用性，配合Nginx作为后端Web服务器，构建高性能、高可用的负载均衡集群。

架构优势

• LVS：内核级负载均衡，性能极高，支持大规模并发
• Keepalived：实现LVS主备高可用，自动故障切换
• Nginx：轻量级高性能Web服务器，作为后端真实服务器
• DR模式：直接路由模式，响应流量不经过LVS，性能最优

二、环境规划

服务器清单

注意：VIP（172.16.1.104）为虚拟IP，不实际分配给任何物理服务器，由Keepalived动态管理。

网络拓扑

┌─────────────────┐
│ Client │
└────────┬────────┘
│
┌────────▼────────┐
│ VIP: 172.16.1.104 │
└────────┬────────┘
│
┌──────────────┼──────────────┐
│ │ │
┌────▼────┐ ┌─────▼─────┐ ┌────▼────┐
│ Master │ │ Backup │ │ Backup │
│ LVS │ │ LVS │ │ LVS │
│ 172.16.1.100│ │172.16.1.101│ │172.16.1.102│
└────┬────┘ └─────┬─────┘ └────┬────┘
│ │ │
└──────────────┼──────────────┘
│
┌────────▼────────┐
│ Real Servers │
│ 172.16.1.102 │
│ 172.16.1.103 │
└─────────────────┘

三、部署Nginx Web服务器

以下操作在 web1 (172.16.1.102) 和 web2 (172.16.1.103) 上执行。

3.1 下载Nginx源码包

wget -O /usr/local/src/nginx-1.31.1.tar.gz https://nginx.org/download/nginx-1.31.1.tar.gz

3.2 安装依赖包

yum install -y vim lrzsz tree screen psmisc lsof tcpdump wget ntpdate \\
gcc gcc-c++ glibc glibc-devel pcre pcre-devel openssl openssl-devel \\
systemd-devel net-tools iotop bc zip unzip zlib-devel bash-completion \\
nfs-utils automake libxml2 libxml2-devel libxslt libxslt-devel \\
perl perl-ExtUtils-Embed

3.3 编译安装Nginx

cd /usr/local/src
tar zxvf nginx-1.31.1.tar.gz
cd nginx-1.31.1/

./configure –prefix=/usr/local/nginx \\
–user=nginx \\
–group=nginx \\
–with-http_ssl_module \\
–with-http_v2_module \\
–with-http_realip_module \\
–with-http_stub_status_module \\
–with-http_gzip_static_module \\
–with-pcre \\
–with-stream \\
–with-stream_ssl_module \\
–with-stream_realip_module

make && make install

3.4 创建Nginx用户

useradd nginx -s /sbin/nologin -u 2000
chown -R nginx:nginx /usr/local/nginx/

3.5 创建Nginx Systemd服务

cat > /usr/lib/systemd/system/nginx.service << 'EOF'
[Unit]
Description=The nginx HTTP and reverse proxy server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStartPre=/usr/bin/rm -f /var/run/nginx.pid
ExecStartPre=/usr/local/nginx/sbin/nginx -t
ExecStart=/usr/local/nginx/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
KillSignal=SIGQUIT
TimeoutStopSec=5
KillMode=process
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF

3.6 创建命令软链接

ln -sv /usr/local/nginx/sbin/nginx /usr/sbin/nginx

3.7 配置Nginx

vim /usr/local/nginx/conf/nginx.conf

user nginx;
worker_processes auto;

error_log /usr/local/nginx/logs/error.log warn;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
}

http {
include /usr/local/nginx/conf/mime.types;
default_type application/octet-stream;
sendfile on;
server_tokens off;
tcp_nopush on;
tcp_nodelay on;

log_format main '$remote_addr – $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log /usr/local/nginx/logs/access.log main;
keepalive_timeout 1d;

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header x-client-ip $remote_addr;

include /usr/local/nginx/conf/conf.d/*.conf;
}

3.8 创建虚拟主机配置

mkdir /usr/local/nginx/conf/conf.d
vim /usr/local/nginx/conf/conf.d/default.conf

server {
listen 80;
server_name localhost;
access_log logs/access.log main;

location / {
root /var/www/html;
index index.html index.htm;
}
}

3.9 创建测试页面

mkdir -p /var/www/html

web1 (172.16.1.102):

echo "Welcome to Nginx Web Server 01" > /var/www/html/index.html

web2 (172.16.1.103):

echo "Welcome to Nginx Web Server 02" > /var/www/html/index.html

3.10 启动Nginx服务

systemctl daemon-reload
systemctl start nginx
systemctl enable nginx
systemctl status nginx

3.11 验证Nginx

# 检查进程
ps -ef | grep nginx

# 查看日志
journalctl -u nginx

# 测试访问
curl http://localhost

3.12 配置防火墙

firewall-cmd –permanent –add-port=80/tcp
firewall-cmd –permanent –add-port=443/tcp
firewall-cmd –reload
firewall-cmd –list-all

四、部署LVS + Keepalived

以下操作在 LVS Master (172.16.1.100) 和 LVS Backup (172.16.1.101) 上执行。

4.1 开启路由转发功能

vim /etc/sysctl.conf

添加以下内容：

net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0 # 根据实际网卡名称调整

使配置生效：

sysctl -p

4.2 安装IPVS

yum -y install ipvsadm

# 加载ip_vs模块
ipvsadm

# 验证模块加载
lsmod | grep ip_vs

4.3 安装Keepalived

yum -y install keepalived

4.4 配置LVS Master (172.16.1.100)

vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
router_id LVS_MASTER
# 注释以下参数，否则可能导致VIP无法正常访问
# vrrp_skip_check_adv_addr
# vrrp_strict
# vrrp_garp_interval 0
# vrrp_gna_interval 0
}

vrrp_instance VI_1 {
state MASTER
interface ens33 # 根据实际网卡名称调整
virtual_router_id 51
priority 100 # MASTER优先级高于BACKUP
advert_int 1
authentication {
auth_type PASS
auth_pass 1111 # 认证密码，主备需一致
}
virtual_ipaddress {
172.16.1.104/24 dev ens33 label ens33:1 # VIP配置
}
}

virtual_server 172.16.1.104 80 {
delay_loop 6
lb_algo rr # 轮询调度算法
lb_kind DR # 直接路由模式
persistence_timeout 50
protocol TCP

real_server 172.16.1.102 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}

real_server 172.16.1.103 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}

4.5 配置LVS Backup (172.16.1.101)

vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
router_id LVS_BACKUP
# vrrp_skip_check_adv_addr
# vrrp_strict
# vrrp_garp_interval 0
# vrrp_gna_interval 0
}

vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 80 # BACKUP优先级低于MASTER
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.1.104/24 dev ens33 label ens33:1
}
}

virtual_server 172.16.1.104 80 {
delay_loop 6
lb_algo rr
lb_kind DR
persistence_timeout 50
protocol TCP

real_server 172.16.1.102 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}

real_server 172.16.1.103 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}

4.6 配置Real Server

在 web1 (172.16.1.102) 和 web2 (172.16.1.103) 上执行以下操作：

vim /etc/rc.d/init.d/realserver.sh

#!/bin/bash
SNS_VIP=172.16.1.104

case "$1" in
start)
ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP
/sbin/route add -host $SNS_VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null 2>&1
echo "RealServer Start OK"
;;
stop)
ifconfig lo:0 down
route del $SNS_VIP >/dev/null 2>&1
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "RealServer Stoped"
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0

添加执行权限并启动：

chmod u+x /etc/rc.d/init.d/realserver.sh
/etc/rc.d/init.d/realserver.sh start

4.7 启动Keepalived服务

# 先启动防火墙
systemctl start firewalld

# 启动Keepalived
systemctl start keepalived
systemctl enable keepalived

# 检查状态
systemctl status keepalived
ps -ef | grep keepalived

重要提示：启动顺序为 先启动防火墙 → 启动Keepalived → 再关闭防火墙（如需关闭）。

4.8 验证VIP分配

LVS Master节点：

ip add | grep ens33
# 应能看到 ens33:1 绑定了 VIP 172.16.1.104

LVS Backup节点：

ip add | grep ens33
# 正常情况下不应看到 VIP

4.9 查看LVS转发规则

ipvsadm -L -n

输出示例：

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.1.104:80 rr persistent 50
-> 172.16.1.102:80 Route 1 0 0
-> 172.16.1.103:80 Route 1 0 0

4.10 实时监控LVS连接

watch -n 1 'ipvsadm -L -n'

五、配置防火墙规则

5.1 LVS服务器防火墙配置

在 LVS Master (172.16.1.100) 和 LVS Backup (172.16.1.101) 上执行：

# 允许VRRP协议
firewall-cmd –direct –permanent –add-rule ipv4 filter INPUT 0 \\
–in-interface ens33 –destination 224.0.0.18 –protocol vrrp -j ACCEPT

firewall-cmd –direct –permanent –add-rule ipv4 filter OUTPUT 0 \\
–out-interface ens33 –destination 224.0.0.18 –protocol vrrp -j ACCEPT

# 开放80端口
firewall-cmd –zone=public –add-port=80/tcp –permanent

# 重载防火墙
firewall-cmd –reload

# 查看配置
iptables -L OUTPUT_direct –line-numbers
iptables -L INPUT_direct –line-numbers

5.2 Web服务器防火墙配置

在 web1 (172.16.1.102) 和 web2 (172.16.1.103) 上执行：

firewall-cmd –zone=public –add-port=80/tcp –permanent
firewall-cmd –reload

5.3 删除防火墙规则（如需要）

firewall-cmd –direct –permanent –remove-rule ipv4 filter INPUT 0 \\
–in-interface ens33 –destination 224.0.0.18 –protocol vrrp -j ACCEPT

firewall-cmd –direct –permanent –remove-rule ipv4 filter OUTPUT 0 \\
–out-interface ens33 –destination 224.0.0.18 –protocol vrrp -j ACCEPT

firewall-cmd –zone=public –remove-port=80/tcp –permanent
firewall-cmd –reload

六、高可用故障切换测试

6.1 测试场景

6.2 测试命令

VIP漂移测试：

# 在Master节点停止Keepalived
systemctl stop keepalived

# 查看Backup节点是否接管VIP
# 在Backup节点执行
ip add | grep 172.16.1.104

# 恢复Master节点
systemctl start keepalived

后端服务器健康检查：

# 停止web1 Nginx
systemctl stop nginx

# 在LVS节点查看转发规则
ipvsadm -L -n
# web1应已被移除

# 恢复web1 Nginx
systemctl start nginx
# web1应重新加入调度池

6.3 验证访问

# 多次访问VIP，验证负载均衡
for i in {1..10}; do
curl http://172.16.1.104
done

预期输出应交替显示 “Welcome to Nginx Web Server 01” 和 “Welcome to Nginx Web Server 02”。

七、LVS调度算法说明

八、DR模式特点与限制

优点

• 高性能：响应流量直接返回客户端，不经过LVS
• 低延迟：减少网络路径
• 扩展性强：后端服务器数量可灵活扩展

限制

解决方案

• 如需不同端口，可使用NAT模式或FULLNAT模式
• 如需跨网段，可使用TUN模式（IP隧道）

九、清理LVS规则

# 删除特定虚拟服务器
ipvsadm -D -t 172.16.1.104:80

# 删除所有规则（慎用）
ipvsadm -C

十、故障排查

10.1 VIP无法访问

# 检查Keepalived状态
systemctl status keepalived

# 查看系统日志
tail -f /var/log/messages

# 检查VIP是否绑定
ip add | grep 172.16.1.104

# 检查防火墙规则
iptables -L -n

10.2 后端无法转发

# 检查LVS转发规则
ipvsadm -L -n

# 检查Real Server配置
/etc/rc.d/init.d/realserver.sh start

# 检查网络连通性
ping 172.16.1.102
curl http://172.16.1.102

10.3 Keepalived不工作

# 检查认证密码是否一致
cat /etc/keepalived/keepalived.conf | grep auth_pass

# 检查VRRP组ID是否一致
cat /etc/keepalived/keepalived.conf | grep virtual_router_id

# 检查网卡名称是否正确
ip link show

十一、生产环境优化建议

十二、总结

通过本次部署，我们成功搭建了基于 LVS + Keepalived + Nginx 的高性能负载均衡集群：

• 高可用：Keepalived实现主备自动切换，故障恢复时间在秒级
• 高性能：LVS工作在内核态，转发效率极高
• 负载均衡：轮询算法将请求均匀分配到后端Web服务器
• 健康检查：自动检测后端服务状态，剔除故障节点

该架构适用于高并发Web服务场景，如门户网站、电商平台、API网关等，是生产环境中广泛使用的成熟方案。如需更高可用性和扩展性，可考虑增加更多LVS节点（多活部署）或结合DNS负载均衡。