欢迎光临
我们一直在努力

Prometheus + Grafana + Alertmanager 高可用监控系统从零搭建指南

📌 前言

本文将从零开始,手把手教您搭建一套生产级高可用的 Prometheus 监控系统,包含:

  • Prometheus 高可用集群(3节点)
  • Node Exporter 节点监控(3节点)
  • Grafana 可视化展示
  • Alertmanager 高可用集群(3节点)+ 告警测试

📋 环境准备

服务器规划

主机名IP 地址部署组件
a1 192.168.194.12 Prometheus + Node Exporter + Alertmanager + Grafana
a2 192.168.194.13 Prometheus + Node Exporter + Alertmanager
a3 192.168.194.14 Prometheus + Node Exporter + Alertmanager

软件版本

组件版本
Prometheus v3.5.3
Node Exporter v1.11.1
Alertmanager v0.33.0
Grafana v13.0.2
操作系统 CentOS 7 / Rocky Linux 8+

网络要求

  • 三台服务器之间网络互通
  • 开放端口:
    • 9090:Prometheus
    • 9100:Node Exporter
    • 9093:Alertmanager
    • 9094:Alertmanager 集群通信
    • 3000:Grafana

一、安装 Prometheus(三台节点)

1.1 下载安装

# 创建监控目录
mkdir -p /opt/monitor

# 下载 Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v3.5.3/prometheus-3.5.3.linux-amd64.tar.gz

# 解压并重命名
tar -xvf prometheus-3.5.3.linux-amd64.tar.gz -C /opt/monitor
mv /opt/monitor/prometheus-3.5.3.linux-amd64 /opt/monitor/prometheus

# 创建专用用户
useradd -r -M -s /sbin/nologin prometheus

# 授权
chown -R prometheus: /opt/monitor/prometheus

1.2 创建 Systemd 服务

cat > /usr/lib/systemd/system/prometheus.service <<'EOF'
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/opt/monitor/prometheus/prometheus \\
–config.file=/opt/monitor/prometheus/prometheus.yml \\
–storage.tsdb.path=/opt/monitor/prometheus/data \\
–storage.tsdb.retention.time=60d \\
–web.listen-address=0.0.0.0:9090 \\
–web.enable-lifecycle
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

1.3 启动 Prometheus

systemctl daemon-reload
systemctl enable prometheus –now
systemctl status prometheus


二、安装 Node Exporter(三台节点)

2.1 下载安装

# 下载
wget https://github.com/prometheus/node_exporter/releases/download/v1.11.1/node_exporter-1.11.1.linux-amd64.tar.gz

# 解压
tar -zxf node_exporter-1.11.1.linux-amd64.tar.gz -C /opt/monitor
mv /opt/monitor/node_exporter-1.11.1.linux-amd64 /opt/monitor/node_exporter

# 授权
chown -R prometheus: /opt/monitor/node_exporter

2.2 创建 Systemd 服务

cat > /usr/lib/systemd/system/node_exporter.service <<'EOF'
[Unit]
Description=node_exporter
Documentation=https://prometheus.io/
After=network.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/opt/monitor/node_exporter/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

2.3 启动 Node Exporter

systemctl daemon-reload
systemctl enable node_exporter –now
systemctl status node_exporter


三、安装 Alertmanager(三台节点)

3.1 下载安装

# 下载
wget https://github.com/prometheus/alertmanager/releases/download/v0.33.0/alertmanager-0.33.0.linux-amd64.tar.gz

# 解压
tar -zxf alertmanager-0.33.0.linux-amd64.tar.gz -C /opt/monitor/
mv /opt/monitor/alertmanager-0.33.0.linux-amd64 /opt/monitor/alertmanager

# 授权
chown -R prometheus: /opt/monitor/alertmanager

3.2 配置文件

默认配置文件 /opt/monitor/alertmanager/alertmanager.yml:

route:
group_by: ['alertname']
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: 'web.hook'

receivers:
– name: 'web.hook'
webhook_configs:
– url: 'http://127.0.0.1:5001/'

inhibit_rules:
– source_matchers: [severity="critical"]
target_matchers: [severity="warning"]
equal: [alertname, dev, instance]

3.3 创建 Systemd 服务

节点 a1(192.168.194.12):

cat > /usr/lib/systemd/system/alertmanager.service <<'EOF'
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/opt/monitor/alertmanager/alertmanager \\
–config.file=/opt/monitor/alertmanager/alertmanager.yml \\
–storage.path=/opt/monitor/alertmanager/data \\
–web.external-url=http://192.168.194.12:9093 \\
–cluster.listen-address=0.0.0.0:9094
Restart=always

[Install]
WantedBy=multi-user.target
EOF

节点 a2(192.168.194.13):

cat > /usr/lib/systemd/system/alertmanager.service <<'EOF'
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/opt/monitor/alertmanager/alertmanager \\
–config.file=/opt/monitor/alertmanager/alertmanager.yml \\
–storage.path=/opt/monitor/alertmanager/data \\
–web.external-url=http://192.168.194.13:9093 \\
–cluster.listen-address=192.168.194.13:9094 \\
–cluster.peer=192.168.194.12:9094
Restart=always

[Install]
WantedBy=multi-user.target
EOF

节点 a3(192.168.194.14):

cat > /usr/lib/systemd/system/alertmanager.service <<'EOF'
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/opt/monitor/alertmanager/alertmanager \\
–config.file=/opt/monitor/alertmanager/alertmanager.yml \\
–storage.path=/opt/monitor/alertmanager/data \\
–web.external-url=http://192.168.194.14:9093 \\
–cluster.listen-address=192.168.194.14:9094 \\
–cluster.peer=192.168.194.12:9094
Restart=always

[Install]
WantedBy=multi-user.target
EOF

3.4 启动 Alertmanager

systemctl daemon-reload
systemctl enable alertmanager –now
systemctl status alertmanager


四、安装 Grafana(仅 a1)

4.1 下载安装

# 下载
wget https://dl.grafana.com/grafana-enterprise/release/13.0.2/grafana-enterprise_13.0.2_26816849631_linux_amd64.tar.gz

# 解压
tar -zxvf grafana-enterprise_13.0.2_26816849631_linux_amd64.tar.gz -C /opt/monitor/
mkdir -p /opt/monitor/grafana
mv /opt/monitor/grafana-13.0.2/* /opt/monitor/grafana/

# 授权
chown -R prometheus: /opt/monitor/grafana

4.2 创建 Systemd 服务

cat > /usr/lib/systemd/system/grafana.service <<'EOF'
[Unit]
Description=Grafana Server
After=network.target

[Service]
Type=simple
User=prometheus
Group=prometheus
Restart=on-failure
ExecStart=/opt/monitor/grafana/bin/grafana server \\
–config=/opt/monitor/grafana/conf/defaults.ini \\
–homepath=/opt/monitor/grafana

[Install]
WantedBy=multi-user.target
EOF

4.3 启动 Grafana

systemctl daemon-reload
systemctl enable grafana –now
systemctl status grafana


五、配置 Prometheus(核心步骤)

5.1 主配置文件 /opt/monitor/prometheus/prometheus.yml

# 全局配置
global:
scrape_interval: 15s # 每15秒采集一次数据
evaluation_interval: 15s # 每15秒评估一次告警规则

# Alertmanager 配置(三个节点)
alerting:
alertmanagers:
– static_configs:
– targets:
– '192.168.194.12:9093'
– '192.168.194.13:9093'
– '192.168.194.14:9093'

# 告警规则文件(后续添加)
rule_files:
# – "alert.yml"

# 采集配置
scrape_configs:
# 监控 Prometheus 自身(三节点)
– job_name: "prometheus"
static_configs:
– targets:
– '192.168.194.12:9090'
– '192.168.194.13:9090'
– '192.168.194.14:9090'

# 监控 Node Exporter(三节点)
– job_name: 'node-exporter'
static_configs:
– targets:
– '192.168.194.12:9100'
– '192.168.194.13:9100'
– '192.168.194.14:9100'
labels:
instance: "node-exporter"

# 监控 Alertmanager(三节点)- 可选
– job_name: 'alertmanager'
static_configs:
– targets:
– '192.168.194.12:9093'
– '192.168.194.13:9093'
– '192.168.194.14:9093'

5.2 验证并重新加载配置

# 验证配置文件语法
/opt/monitor/prometheus/promtool check config /opt/monitor/prometheus/prometheus.yml

# 热加载(无需重启)
curl -X POST http://localhost:9090/-/reload

# 或重启服务
systemctl restart prometheus

5.3 同步配置文件到其他节点

# 将 a1 的配置同步到 a2、a3
scp /opt/monitor/prometheus/prometheus.yml root@192.168.194.13:/opt/monitor/prometheus/
scp /opt/monitor/prometheus/prometheus.yml root@192.168.194.14:/opt/monitor/prometheus/

# 在 a2、a3 上分别 reload
curl -X POST http://192.168.194.13:9090/-/reload
curl -X POST http://192.168.194.14:9090/-/reload


六、验证与测试

6.1 访问 Prometheus Targets

浏览器打开:http://192.168.194.12:9090/targets

应该看到所有目标状态为 UP:

Job数量状态
prometheus 3/3 ✅ UP
node-exporter 3/3 ✅ UP
alertmanager 3/3 ✅ UP

在这里插入图片描述

6.2 测试 Alertmanager

发送测试告警(v2 API):

curl -X POST http://localhost:9093/api/v2/alerts \\
-H "Content-Type: application/json" \\
-d '[
{
"labels": {
"alertname": "TestAlert",
"severity": "warning",
"instance": "192.168.194.12"
},
"annotations": {
"summary": "测试告警",
"description": "这是一个从命令行发送的测试告警"
}
}
]'

验证告警是否到达:

浏览器打开:http://192.168.194.12:9093/#/alerts

在这里插入图片描述

6.3 验证 Alertmanager 高可用

分别在三个节点发送测试告警:

# a1
curl -X POST http://192.168.194.12:9093/api/v2/alerts \\
-H "Content-Type: application/json" \\
-d '[{"labels":{"alertname":"TestAlert","instance":"a1"},"annotations":{"summary":"Test from a1"}}]'

# a2
curl -X POST http://192.168.194.13:9093/api/v2/alerts \\
-H "Content-Type: application/json" \\
-d '[{"labels":{"alertname":"TestAlert","instance":"a2"},"annotations":{"summary":"Test from a2"}}]'

# a3
curl -X POST http://192.168.194.14:9093/api/v2/alerts \\
-H "Content-Type: application/json" \\
-d '[{"labels":{"alertname":"TestAlert","instance":"a3"},"annotations":{"summary":"Test from a3"}}]'

访问任意节点的 /api/v2/alerts 查看所有告警是否同步。

6.4 查看 Alertmanager 集群状态

访问:http://192.168.194.12:9093/#/status

应该看到 3 个节点都在集群中。 在这里插入图片描述

6.5 访问 Grafana

浏览器打开:http://192.168.194.12:3000

  • 默认账号:admin
  • 默认密码:admin

添加 Prometheus 数据源:http://192.168.194.12:9090 在这里插入图片描述 在这里插入图片描述

在这里插入图片描述 在这里插入图片描述


七、系统架构图

┌─────────────────────────────────────────────┐
│ 用户访问 │
└─────────────────────────────────────────────┘
│
┌──────────────────┴──────────────────┐
│ Grafana (a1) │
│ http://:3000 │
└──────────────────┬──────────────────┘
│
┌──────────────────┴──────────────────┐
│ Prometheus 高可用集群 │
├─────────────────────────────────────┤
│ a1:9090 │ a2:9090 │ a3:9090 │
└─────────────────────────────────────┘
│ │ │
┌──────┴──────┬────┴────┬─────┴──────┐
│ │ │ │
┌─────┴─────┐ ┌─────┴─────┐ ┌─────┴─────┐
│ Node Exp │ │ Node Exp │ │ Node Exp │
│ a1:9100 │ │ a2:9100 │ │ a3:9100 │
└───────────┘ └───────────┘ └───────────┘

┌─────────────────────────────────────┐
│ Alertmanager 高可用集群 │
├─────────────────────────────────────┤
│ a1:9093 │ a2:9093 │ a3:9093 │
│ (集群) │ (集群) │ (集群) │
└───────────┴───────────┴───────────┘


八、常用运维命令

服务管理

# 启动/停止/重启/状态
systemctl start|stop|restart|status prometheus
systemctl start|stop|restart|status node_exporter
systemctl start|stop|restart|status alertmanager
systemctl start|stop|restart|status grafana

查看日志

journalctl -u prometheus -f
journalctl -u alertmanager -f -n 50

配置文件验证

# Prometheus
/opt/monitor/prometheus/promtool check config /opt/monitor/prometheus/prometheus.yml

# Alertmanager
/opt/monitor/alertmanager/amtool check-config /opt/monitor/alertmanager/alertmanager.yml

热加载配置

# Prometheus
curl -X POST http://localhost:9090/-/reload

# Alertmanager
curl -X POST http://localhost:9093/-/reload


九、常见问题排查

9.1 Alertmanager v1 API 弃用

问题:使用 v1 API 报错

{"status":"deprecated","error":"The Alertmanager v1 API was deprecated…"}

解决:使用 v2 API 并添加 Content-Type: application/json

9.2 Targets 显示 DOWN

排查步骤:

  • 检查目标服务是否运行:systemctl status node_exporter
  • 检查防火墙:firewall-cmd –list-ports
  • 检查网络连通性:telnet 192.168.194.13 9100
  • 查看 Prometheus 日志:journalctl -u prometheus -f
  • 9.3 Alertmanager 集群未生效

    排查步骤:

  • 检查 –cluster.peer 参数是否正确
  • 检查 9094 端口是否开放
  • 查看集群状态:访问 http://:9093/#/status

  • 十、总结

    至此,您已经完成了一套 3节点高可用 Prometheus 监控系统 的搭建,包含:

    组件高可用方式作用
    Prometheus 3节点独立运行,Grafana 可配置多数据源 指标采集与存储
    Node Exporter 每台机器部署 系统级指标采集
    Alertmanager 3节点集群(Gossip 协议) 告警去重、分组、路由
    Grafana 单节点 数据可视化

    📚 参考链接

    • Prometheus 官网
    • Prometheus 下载
    • Grafana 官网
    • Alertmanager 文档
    赞(0)
    未经允许不得转载:171主机测评 » Prometheus + Grafana + Alertmanager 高可用监控系统从零搭建指南
    分享到: 更多 (0)

    评论 抢沙发

    • 昵称 (必填)
    • 邮箱 (必填)
    • 网址