📌 前言
本文将从零开始,手把手教您搭建一套生产级高可用的 Prometheus 监控系统,包含:
- Prometheus 高可用集群(3节点)
- Node Exporter 节点监控(3节点)
- Grafana 可视化展示
- Alertmanager 高可用集群(3节点)+ 告警测试
📋 环境准备
服务器规划
| a1 | 192.168.194.12 | Prometheus + Node Exporter + Alertmanager + Grafana |
| a2 | 192.168.194.13 | Prometheus + Node Exporter + Alertmanager |
| a3 | 192.168.194.14 | Prometheus + Node Exporter + Alertmanager |
软件版本
| Prometheus | v3.5.3 |
| Node Exporter | v1.11.1 |
| Alertmanager | v0.33.0 |
| Grafana | v13.0.2 |
| 操作系统 | CentOS 7 / Rocky Linux 8+ |
网络要求
- 三台服务器之间网络互通
- 开放端口:
- 9090:Prometheus
- 9100:Node Exporter
- 9093:Alertmanager
- 9094:Alertmanager 集群通信
- 3000:Grafana
一、安装 Prometheus(三台节点)
1.1 下载安装
# 创建监控目录
mkdir -p /opt/monitor
# 下载 Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v3.5.3/prometheus-3.5.3.linux-amd64.tar.gz
# 解压并重命名
tar -xvf prometheus-3.5.3.linux-amd64.tar.gz -C /opt/monitor
mv /opt/monitor/prometheus-3.5.3.linux-amd64 /opt/monitor/prometheus
# 创建专用用户
useradd -r -M -s /sbin/nologin prometheus
# 授权
chown -R prometheus: /opt/monitor/prometheus
1.2 创建 Systemd 服务
cat > /usr/lib/systemd/system/prometheus.service <<'EOF'
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/opt/monitor/prometheus/prometheus \\
–config.file=/opt/monitor/prometheus/prometheus.yml \\
–storage.tsdb.path=/opt/monitor/prometheus/data \\
–storage.tsdb.retention.time=60d \\
–web.listen-address=0.0.0.0:9090 \\
–web.enable-lifecycle
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
1.3 启动 Prometheus
systemctl daemon-reload
systemctl enable prometheus –now
systemctl status prometheus
二、安装 Node Exporter(三台节点)
2.1 下载安装
# 下载
wget https://github.com/prometheus/node_exporter/releases/download/v1.11.1/node_exporter-1.11.1.linux-amd64.tar.gz
# 解压
tar -zxf node_exporter-1.11.1.linux-amd64.tar.gz -C /opt/monitor
mv /opt/monitor/node_exporter-1.11.1.linux-amd64 /opt/monitor/node_exporter
# 授权
chown -R prometheus: /opt/monitor/node_exporter
2.2 创建 Systemd 服务
cat > /usr/lib/systemd/system/node_exporter.service <<'EOF'
[Unit]
Description=node_exporter
Documentation=https://prometheus.io/
After=network.target
[Service]
User=prometheus
Group=prometheus
ExecStart=/opt/monitor/node_exporter/node_exporter
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
2.3 启动 Node Exporter
systemctl daemon-reload
systemctl enable node_exporter –now
systemctl status node_exporter
三、安装 Alertmanager(三台节点)
3.1 下载安装
# 下载
wget https://github.com/prometheus/alertmanager/releases/download/v0.33.0/alertmanager-0.33.0.linux-amd64.tar.gz
# 解压
tar -zxf alertmanager-0.33.0.linux-amd64.tar.gz -C /opt/monitor/
mv /opt/monitor/alertmanager-0.33.0.linux-amd64 /opt/monitor/alertmanager
# 授权
chown -R prometheus: /opt/monitor/alertmanager
3.2 配置文件
默认配置文件 /opt/monitor/alertmanager/alertmanager.yml:
route:
group_by: ['alertname']
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: 'web.hook'
receivers:
– name: 'web.hook'
webhook_configs:
– url: 'http://127.0.0.1:5001/'
inhibit_rules:
– source_matchers: [severity="critical"]
target_matchers: [severity="warning"]
equal: [alertname, dev, instance]
3.3 创建 Systemd 服务
节点 a1(192.168.194.12):
cat > /usr/lib/systemd/system/alertmanager.service <<'EOF'
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/opt/monitor/alertmanager/alertmanager \\
–config.file=/opt/monitor/alertmanager/alertmanager.yml \\
–storage.path=/opt/monitor/alertmanager/data \\
–web.external-url=http://192.168.194.12:9093 \\
–cluster.listen-address=0.0.0.0:9094
Restart=always
[Install]
WantedBy=multi-user.target
EOF
节点 a2(192.168.194.13):
cat > /usr/lib/systemd/system/alertmanager.service <<'EOF'
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/opt/monitor/alertmanager/alertmanager \\
–config.file=/opt/monitor/alertmanager/alertmanager.yml \\
–storage.path=/opt/monitor/alertmanager/data \\
–web.external-url=http://192.168.194.13:9093 \\
–cluster.listen-address=192.168.194.13:9094 \\
–cluster.peer=192.168.194.12:9094
Restart=always
[Install]
WantedBy=multi-user.target
EOF
节点 a3(192.168.194.14):
cat > /usr/lib/systemd/system/alertmanager.service <<'EOF'
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/opt/monitor/alertmanager/alertmanager \\
–config.file=/opt/monitor/alertmanager/alertmanager.yml \\
–storage.path=/opt/monitor/alertmanager/data \\
–web.external-url=http://192.168.194.14:9093 \\
–cluster.listen-address=192.168.194.14:9094 \\
–cluster.peer=192.168.194.12:9094
Restart=always
[Install]
WantedBy=multi-user.target
EOF
3.4 启动 Alertmanager
systemctl daemon-reload
systemctl enable alertmanager –now
systemctl status alertmanager
四、安装 Grafana(仅 a1)
4.1 下载安装
# 下载
wget https://dl.grafana.com/grafana-enterprise/release/13.0.2/grafana-enterprise_13.0.2_26816849631_linux_amd64.tar.gz
# 解压
tar -zxvf grafana-enterprise_13.0.2_26816849631_linux_amd64.tar.gz -C /opt/monitor/
mkdir -p /opt/monitor/grafana
mv /opt/monitor/grafana-13.0.2/* /opt/monitor/grafana/
# 授权
chown -R prometheus: /opt/monitor/grafana
4.2 创建 Systemd 服务
cat > /usr/lib/systemd/system/grafana.service <<'EOF'
[Unit]
Description=Grafana Server
After=network.target
[Service]
Type=simple
User=prometheus
Group=prometheus
Restart=on-failure
ExecStart=/opt/monitor/grafana/bin/grafana server \\
–config=/opt/monitor/grafana/conf/defaults.ini \\
–homepath=/opt/monitor/grafana
[Install]
WantedBy=multi-user.target
EOF
4.3 启动 Grafana
systemctl daemon-reload
systemctl enable grafana –now
systemctl status grafana
五、配置 Prometheus(核心步骤)
5.1 主配置文件 /opt/monitor/prometheus/prometheus.yml
# 全局配置
global:
scrape_interval: 15s # 每15秒采集一次数据
evaluation_interval: 15s # 每15秒评估一次告警规则
# Alertmanager 配置(三个节点)
alerting:
alertmanagers:
– static_configs:
– targets:
– '192.168.194.12:9093'
– '192.168.194.13:9093'
– '192.168.194.14:9093'
# 告警规则文件(后续添加)
rule_files:
# – "alert.yml"
# 采集配置
scrape_configs:
# 监控 Prometheus 自身(三节点)
– job_name: "prometheus"
static_configs:
– targets:
– '192.168.194.12:9090'
– '192.168.194.13:9090'
– '192.168.194.14:9090'
# 监控 Node Exporter(三节点)
– job_name: 'node-exporter'
static_configs:
– targets:
– '192.168.194.12:9100'
– '192.168.194.13:9100'
– '192.168.194.14:9100'
labels:
instance: "node-exporter"
# 监控 Alertmanager(三节点)- 可选
– job_name: 'alertmanager'
static_configs:
– targets:
– '192.168.194.12:9093'
– '192.168.194.13:9093'
– '192.168.194.14:9093'
5.2 验证并重新加载配置
# 验证配置文件语法
/opt/monitor/prometheus/promtool check config /opt/monitor/prometheus/prometheus.yml
# 热加载(无需重启)
curl -X POST http://localhost:9090/-/reload
# 或重启服务
systemctl restart prometheus
5.3 同步配置文件到其他节点
# 将 a1 的配置同步到 a2、a3
scp /opt/monitor/prometheus/prometheus.yml root@192.168.194.13:/opt/monitor/prometheus/
scp /opt/monitor/prometheus/prometheus.yml root@192.168.194.14:/opt/monitor/prometheus/
# 在 a2、a3 上分别 reload
curl -X POST http://192.168.194.13:9090/-/reload
curl -X POST http://192.168.194.14:9090/-/reload
六、验证与测试
6.1 访问 Prometheus Targets
浏览器打开:http://192.168.194.12:9090/targets
应该看到所有目标状态为 UP:
| prometheus | 3/3 | ✅ UP |
| node-exporter | 3/3 | ✅ UP |
| alertmanager | 3/3 | ✅ UP |

6.2 测试 Alertmanager
发送测试告警(v2 API):
curl -X POST http://localhost:9093/api/v2/alerts \\
-H "Content-Type: application/json" \\
-d '[
{
"labels": {
"alertname": "TestAlert",
"severity": "warning",
"instance": "192.168.194.12"
},
"annotations": {
"summary": "测试告警",
"description": "这是一个从命令行发送的测试告警"
}
}
]'
验证告警是否到达:
浏览器打开:http://192.168.194.12:9093/#/alerts

6.3 验证 Alertmanager 高可用
分别在三个节点发送测试告警:
# a1
curl -X POST http://192.168.194.12:9093/api/v2/alerts \\
-H "Content-Type: application/json" \\
-d '[{"labels":{"alertname":"TestAlert","instance":"a1"},"annotations":{"summary":"Test from a1"}}]'
# a2
curl -X POST http://192.168.194.13:9093/api/v2/alerts \\
-H "Content-Type: application/json" \\
-d '[{"labels":{"alertname":"TestAlert","instance":"a2"},"annotations":{"summary":"Test from a2"}}]'
# a3
curl -X POST http://192.168.194.14:9093/api/v2/alerts \\
-H "Content-Type: application/json" \\
-d '[{"labels":{"alertname":"TestAlert","instance":"a3"},"annotations":{"summary":"Test from a3"}}]'
访问任意节点的 /api/v2/alerts 查看所有告警是否同步。
6.4 查看 Alertmanager 集群状态
访问:http://192.168.194.12:9093/#/status
应该看到 3 个节点都在集群中。 
6.5 访问 Grafana
浏览器打开:http://192.168.194.12:3000
- 默认账号:admin
- 默认密码:admin
添加 Prometheus 数据源:http://192.168.194.12:9090


七、系统架构图
┌─────────────────────────────────────────────┐
│ 用户访问 │
└─────────────────────────────────────────────┘
│
┌──────────────────┴──────────────────┐
│ Grafana (a1) │
│ http://:3000 │
└──────────────────┬──────────────────┘
│
┌──────────────────┴──────────────────┐
│ Prometheus 高可用集群 │
├─────────────────────────────────────┤
│ a1:9090 │ a2:9090 │ a3:9090 │
└─────────────────────────────────────┘
│ │ │
┌──────┴──────┬────┴────┬─────┴──────┐
│ │ │ │
┌─────┴─────┐ ┌─────┴─────┐ ┌─────┴─────┐
│ Node Exp │ │ Node Exp │ │ Node Exp │
│ a1:9100 │ │ a2:9100 │ │ a3:9100 │
└───────────┘ └───────────┘ └───────────┘
┌─────────────────────────────────────┐
│ Alertmanager 高可用集群 │
├─────────────────────────────────────┤
│ a1:9093 │ a2:9093 │ a3:9093 │
│ (集群) │ (集群) │ (集群) │
└───────────┴───────────┴───────────┘
八、常用运维命令
服务管理
# 启动/停止/重启/状态
systemctl start|stop|restart|status prometheus
systemctl start|stop|restart|status node_exporter
systemctl start|stop|restart|status alertmanager
systemctl start|stop|restart|status grafana
查看日志
journalctl -u prometheus -f
journalctl -u alertmanager -f -n 50
配置文件验证
# Prometheus
/opt/monitor/prometheus/promtool check config /opt/monitor/prometheus/prometheus.yml
# Alertmanager
/opt/monitor/alertmanager/amtool check-config /opt/monitor/alertmanager/alertmanager.yml
热加载配置
# Prometheus
curl -X POST http://localhost:9090/-/reload
# Alertmanager
curl -X POST http://localhost:9093/-/reload
九、常见问题排查
9.1 Alertmanager v1 API 弃用
问题:使用 v1 API 报错
{"status":"deprecated","error":"The Alertmanager v1 API was deprecated…"}
解决:使用 v2 API 并添加 Content-Type: application/json
9.2 Targets 显示 DOWN
排查步骤:
9.3 Alertmanager 集群未生效
排查步骤:
十、总结
至此,您已经完成了一套 3节点高可用 Prometheus 监控系统 的搭建,包含:
| Prometheus | 3节点独立运行,Grafana 可配置多数据源 | 指标采集与存储 |
| Node Exporter | 每台机器部署 | 系统级指标采集 |
| Alertmanager | 3节点集群(Gossip 协议) | 告警去重、分组、路由 |
| Grafana | 单节点 | 数据可视化 |
📚 参考链接
- Prometheus 官网
- Prometheus 下载
- Grafana 官网
- Alertmanager 文档






