Prometheus监控服务器及K8s集群资源

Prometheus+Grafana监控服务器及K8s集群资源

本文档详细介绍了如何在 Rocky Linux 8.5 系统上部署 Prometheus + Grafana，实现对服务器资源及 Kubernetes 集群组件可视化监控，内容包含环境准备、组件安装、配置优化、报警设置等。

一、环境信息概览

K8s部署参考：https://blog.csdn.net/weixin_45867513/article/details/144565165?spm=1011.2415.3001.5331

二、核心组件与端口说明

下载方式：

• prometheus官网：https://prometheus.io/download/

• grafana官网：https://grafana.com/grafana/download?pg=graf&plcmt=deploy-box-1

• 百度网盘（含所有软件包）：https://pan.baidu.com/s/18ekfed1AxZ8BmwO_bJHxNA 提取码: 4hzu

三、环境准备（104）

3.1 修改主机名 & 防火墙设置

[root@localhost ~]# systemctl disable –now firewalld
[root@localhost ~]# hostnamectl set-hostname prometheus && bash

3.2 设置 Hosts 文件

[root@prometheus ~]# cat >> /etc/hosts << EOF
192.168.200.11 k8s-master
192.168.200.12 k8s-node1
192.168.200.13 k8s-node2
192.168.200.14 prometheus
EOF

3.3 免密登录设置

[root@prometheus ~]# ssh-keygen
[root@prometheus ~]# ssh-copy-id prometheus
[root@prometheus ~]# ssh-copy-id k8s-master01
[root@prometheus ~]# ssh-copy-id k8s-node01
[root@prometheus ~]# ssh-copy-id k8s-node02

3.4 时间同步配置

yum install -y chrony
echo "server ntp.aliyun.com iburst" >> /etc/chrony.conf
systemctl enable –now chronyd

3.5 上传软件包

mkdir -pv /data/software
cd /data/software
# 上传所有组件 tar.gz 包至该目录

四、安装 Prometheus 组件

4.1 Prometheus 安装

[root@prometheus software]# tar -xf prometheus-2.29.1.linux-amd64.tar.gz -C /usr/local/

配置文件：/usr/local/prometheus/prometheus.yml修改前备份文件

static_configs: #使用静态配置方式指定采集目标。
– targets: ["localhost:9090"] #Prometheus 自身的监控地址（默认端口 9090）
– job_name: 'pushgateway' #定义任务名称，用于标识该采集任务
static_configs:
– targets: ['192.168.200.14:9091'] # prometheus所在主机IP
labels:
instance: pushgateway
– job_name: 'node exporter'
static_configs:
– targets: ['192.168.200.11:9100','192.168.200.12:9100','192.168.200.13:9100','192.168.200.14:9100'] #被采集的客户端ip，prometheus会从http://这这些ip/metrics拉取数据

这个配置文件主要定义了三类监控

Prometheus 自身：监控 Prometheus 服务的运行状态。

Push Gateway：接收短期作业推送的指标。

Node Exporter 集群：监控多台服务器的系统资源使用情况。

创建 systemd 服务：/etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/prometheus-2.29.1/prometheus –config.file=/usr/local/prometheus-2.29.1/prometheus.yml
WorkingDirectory=/usr/local/prometheus-2.29.1
Restart=always

[Install]
WantedBy=multi-user.target

启动服务：

systemctl daemon-reload
systemctl enable –now prometheus

4.2 PushGateway安装

prometehes拉取数据是比较困难的，所以中间有一个中转站pushgateway，他更像是metric和prometehes的填充，prometehes会到pushgateway上去获取数据

[root@prometheus prometheus-2.29.1]# cd /data/software/
[root@prometheus software]# tar -xf pushgateway-1.4.1.linux-amd64.tar.gz
[root@prometheus software]# mv pushgateway-1.4.1.linux-amd64 /usr/local/pushgateway-1.4.1

创建服务：/etc/systemd/system/pushgateway.service

[Unit]
Description=Pushgateway
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/pushgateway-1.4.1/pushgateway –web.listen-address :9091
WorkingDirectory=/usr/local/pushgateway-1.4.1
Restart=always

[Install]
WantedBy=multi-user.target

启动服务：

systemctl daemon-reload
systemctl enable –now pushgateway

4.3 Alermanager安装

这个软件主要用于报警

[root@prometheus software]# tar -xf alertmanager-0.28.1.linux-amd64.tar.gz
[root@prometheus software]# mv alertmanager-0.28.1.linux-amd64 alertmanager-0.28.1

创建服务：/etc/systemd/system/alertmanager.service

[Unit]
Description=Alertmanager
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/alertmanager-0.28.1/alertmanager –config.file=/usr/local/alertmanager-0.28.1/alertmanager.yml
WorkingDirectory=/usr/local/alertmanager-0.28.1
Restart=always

[Install]
WantedBy=multi-user.target

启动服务：

systemctl daemon-reload
systemctl enable –now alertmanager

4.4 Node Exporter安装（所有节点）

安装到哪个node上就能获取哪个node的节点信息，prometheus通常会从暴露的HTTP服务地址（通常是/metrics）拉取监控样本数据，最终存放在自己时序数据库中

[root@prometheus software]# tar -xf node_exporter-1.2.2.linux-amd64.tar.gz
[root@prometheus software]# mv node_exporter-1.2.2.linux-amd64 /usr/local/node_exporter-1.2.2

远程给其它主机

[root@prometheus software]# cd /usr/local/
[root@prometheus local]# scp -r node_exporter-1.2.2/ k8s-master:/usr/local/
LICENSE 100% 11KB 6.3MB/s 00:00
NOTICE 100% 463 426.7KB/s 00:00
node_exporter 100% 18MB 64.7MB/s 00:00
[root@prometheus local]# scp -r node_exporter-1.2.2/ k8s-node1:/usr/local/
LICENSE 100% 11KB 6.7MB/s 00:00
NOTICE 100% 463 211.1KB/s 00:00
node_exporter 100% 18MB 66.2MB/s 00:00
[root@prometheus local]# scp -r node_exporter-1.2.2/ k8s-node2:/usr/local/
LICENSE 100% 11KB 5.9MB/s 00:00
NOTICE 100% 463 394.6KB/s 00:00
node_exporter 100% 18MB 83.0MB/s 00:00

创建服务：/etc/systemd/system/node_exporter.service

[root@prometheus local]# vim /etc/systemd/system/node_exporter.service

[Unit]
Description=node_export
Documentation=https://github.com/prometheus/node_exporter
After=network.target
[Service]
Type=simple
User=root
ExecStart= /usr/local/node_exporter-1.2.2/node_exporter
Restart=on-failure
[Install]
WantedBy=multi-user.target

远程给其它主机：

#给其他主机传送
[root@prometheus local]# scp -r /etc/systemd/system/node_exporter.service k8s-master:/etc/systemd/system/
node_exporter.service 100% 254 271.2KB/s 00:00
[root@prometheus local]# scp -r /etc/systemd/system/node_exporter.service k8s-node1:/etc/systemd/system/
node_exporter.service 100% 254 199.4KB/s 00:00
[root@prometheus local]# scp -r /etc/systemd/system/node_exporter.service k8s-node2:/etc/systemd/system/
node_exporter.service 100% 254 130.4KB/s 00:00

所有主机启动exporter服务

[root@prometheus local]# systemctl enable –now node_exporter
[root@k8s-master ~]# systemctl enable –now node_exporter
[root@k8s-node1 ~]# systemctl enable –now node_exporter
[root@k8s-node2 ~]# systemctl enable –now node_exporter

所有主机访问9100端口：

192.168.200.11:9100/metrics

192.168.200.12:9100/metrics

192.168.200.13:9100/metricss

192.168.200.14:9100/metrics

能访问该端口，表明目标主机上的 Node Exporter 服务已成功启动并正常监听，正在通过 http://IP:9100/metrics 提供数据供prometheus拉取。

启动prometheus和pushgateway

systemctl daemon-reload
systemctl enable –now prometheus
systemctl enable –now pushgateway

访问prometheus服务端界面

五、Grafana 安装与使用

5.1 Grafana安装

[root@prometheus software]# cd /data/software/
[root@prometheus software]# tar -xf grafana-enterprise-12.0.0.linux-amd64.tar.gz -C /usr/local/

创建Granfana服务单元/etc/systemd/system/grafana.service

[Unit]
Description=Grafana
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/grafana-v12.0.0/bin/grafana-server web
WorkingDirectory=/usr/local/grafana-v12.0.0
Restart=always

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable –now grafana.service

访问地址：http://192.168.200.14:3000（默认账号密码：admin / admin） 首次登录需强制修改密码

5.2 添加数据源

这个是prometheus服务器地址：http://192.168.200.14:9090 输入完成后点击下方的Save&test按钮

5.3 导入模板

点击右上角搜索按钮

点击导入模板

输入模板ID 8919 导入

5.4 图形化数据展示

Grafana官网 https://grafana.com/grafana/dashboards/ 多个模板可供选择

六、Kubernetes 集群监控（kube-state-metrics 安装）

6.1 安装Kube-state-metrics服务

• 作用： 通过监听 Kubernetes API Server，将集群内资源对象（ Pod、Deployment、Service、Namespace、ConfigMap 等）的状态转换为 Prometheus 可采集的时间序列指标。

yaml文件创建kube-state-metrics.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/component: exporter
app.kubernetes.io/name: kube–state–metrics
app.kubernetes.io/version: 1.9.0
name: kube–state–metrics
rules:
– apiGroups:
– ""
resources:
– configmaps
– secrets
– nodes
– pods
– services
– serviceaccounts
– resourcequotas
– replicationcontrollers
– limitranges
– persistentvolumeclaims
– persistentvolumes
– namespaces
– endpoints
verbs:
– list
– watch
– apiGroups:
– apps
resources:
– statefulsets
– daemonsets
– deployments
– replicasets
verbs:
– list
– watch
– apiGroups:
– batch
resources:
– cronjobs
– jobs
verbs:
– list
– watch
– apiGroups:
– autoscaling
resources:
– horizontalpodautoscalers
verbs:
– list
– watch
– apiGroups:
– authentication.k8s.io
resources:
– tokenreviews
verbs:
– create
– apiGroups:
– authorization.k8s.io
resources:
– subjectaccessreviews
verbs:
– create
– apiGroups:
– policy
resources:
– poddisruptionbudgets
verbs:
– list
– watch
– apiGroups:
– certificates.k8s.io
resources:
– certificatesigningrequests
verbs:
– list
– watch
– apiGroups:
– discovery.k8s.io
resources:
– endpointslices
verbs:
– list
– watch
– apiGroups:
– storage.k8s.io
resources:
– storageclasses
– volumeattachments
verbs:
– list
– watch
– apiGroups:
– admissionregistration.k8s.io
resources:
– mutatingwebhookconfigurations
– validatingwebhookconfigurations
verbs:
– list
– watch
– apiGroups:
– networking.k8s.io
resources:
– networkpolicies
– ingressclasses
– ingresses
verbs:
– list
– watch
– apiGroups:
– coordination.k8s.io
resources:
– leases
verbs:
– list
– watch
– apiGroups:
– rbac.authorization.k8s.io
resources:
– clusterrolebindings
– clusterroles
– rolebindings
– roles
verbs:
– list
– watch
—
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/component: exporter
app.kubernetes.io/name: kube–state–metrics
app.kubernetes.io/version: 1.9.0
name: kube–state–metrics
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube–state–metrics
subjects:
– kind: ServiceAccount
name: kube–state–metrics
namespace: kube–system
—
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/component: exporter
app.kubernetes.io/name: kube–state–metrics
app.kubernetes.io/version: 1.9.0
name: kube–state–metrics
namespace: kube–system
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: kube–state–metrics
template:
metadata:
labels:
app.kubernetes.io/component: exporter
app.kubernetes.io/name: kube–state–metrics
app.kubernetes.io/version: 1.9.0
spec:
automountServiceAccountToken: true
containers:
– image: swr.cn–north–4.myhuaweicloud.com/ddn–k8s/registry.k8s.io/kube–state–metrics/kube–state–metrics:v2.9.2
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
timeoutSeconds: 5
name: kube–state–metrics
ports:
– containerPort: 8080
name: http–metrics
– containerPort: 8081
name: telemetry
readinessProbe:
httpGet:
path: /
port: 8081
initialDelaySeconds: 5
timeoutSeconds: 5
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
– ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 65534
seccompProfile:
type: RuntimeDefault
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: kube–state–metrics
—
apiVersion: v1
automountServiceAccountToken: false
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: exporter
app.kubernetes.io/name: kube–state–metrics
app.kubernetes.io/version: 1.9.0
name: kube–state–metrics
namespace: kube–system
—
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: exporter
app.kubernetes.io/name: kube–state–metrics
app.kubernetes.io/version: 1.9.0
name: kube–state–metrics
namespace: kube–system
spec:
type: NodePort
ports:
– name: http–metrics
port: 9090
protocol: TCP
targetPort: http–metrics
nodePort: 31031
– name: telemetry
port: 9091
protocol: TCP
targetPort: telemetry
nodePort: 31032
selector:
app.kubernetes.io/name: kube–state–metrics

kubectl apply -f kube-state-metrics.yaml

#查看pod运行状态
[root@k8s-master01 ~]# kubectl get pod -A |grep kube-state-metrics
kube-system kube-state-metrics-cc7d6998c-kjj4b 1/1 Running 0 20h

[root@k8s-master01 ~]# kubectl get svc -A |grep kube-state-metrics
kube-system kube-state-metrics NodePort 10.100.7.78 <none> 9090:31031/TCP,9091:31032/TCP 20h

#修改prometheus.yml文件添加
– job_name: "kube-state-metrics"
static_configs:
– targets: ['192.168.200.11:31031']

#重启prometheus
systemctl restart prometheus

6.2 模板导入

七、邮箱报警

7.1 Alertmanager邮箱配置

cp /usr/local/alertmanager-0.28.1/alertmanager.yml{,.bak}

vim

编辑 /usr/local/alertmanager-0.28.1/alertmanager.yml

这个文档使用163邮箱报警

global:
smtp_smarthost: 'smtp.163.com:465'
smtp_from: 'jin1205577136@163.com'
smtp_auth_username: 'jin1205577136@163.com'
smtp_auth_password: 'PTbz4G4WEstghuh7'
smtp_require_tls: false

templates:
– '/usr/local/alertmanager-0.28.1/custom.tmpl'

route:
receiver: '邮件接收人'
group_wait: 0s
group_interval: 30s
repeat_interval: 300s

receivers:
– name: '邮件接收人'
email_configs:
– to: 'jin1205577136@163.com' #多个收件人用逗号分隔
send_resolved: true
html: '{{ template "custom_alert_template" . }}'
headers:
Subject: |
{{- if eq .Status "firing" -}}
[⚠️紧急告警] 🚨 触发中：{{ .CommonLabels.alertname }} – 实例: {{ .CommonLabels.instance }}
{{- else -}}
[✅已恢复] 🔔 告警解决：{{ .CommonLabels.alertname }} – 实例: {{ .CommonLabels.instance }}
{{- end -}}

这个文档使用钉钉报警

global:
# 移除 SMTP 相关配置（如果不再使用邮箱）

templates:
– '/usr/local/alertmanager-0.28.1/custom.tmpl' # 只需包含钉钉模板

route:
receiver: 'dingtalk-notification' # 默认接收器改为钉钉
group_wait: 0s
group_interval: 30s
repeat_interval: 300s

receivers:
– name: 'dingtalk-notification'
webhook_configs:
– url: 'https://oapi.dingtalk.com/robot/send?access_token=xxx' # 替换为你的钉钉 Webhook
send_resolved: true
http_config:
tls_config:
insecure_skip_verify: true
headers:
Content-Type: 'application/json'
body: '{{ template "dingtalk_alert_template" . }}' # 引用钉钉模板

7.2 邮箱模板配置

**注意：**以下为邮箱和钉钉两种方案，选择其中一个即可。

**方案一：**邮箱配置

#创建模板
cd /usr/local/alertmanager-0.28.1/
vim custom.tmpl

{{ define "custom_alert_template" }}
{{- if eq .Status "firing" -}}
<div style="max-width:750px; margin:0 auto; font-family:'Inter',-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,'Helvetica Neue',Arial,sans-serif; background-color:#f9fafb; border-radius:12px; overflow:hidden; box-shadow:0 10px 25px -5px rgba(0,0,0,0.1),0 8px 10px -6px rgba(0,0,0,0.1);">
<div style="background:linear-gradient(135deg, #e53935 0%, #b71c1c 100%); color:white; padding:25px 30px; position:relative; overflow:hidden;">
<div style="position:absolute; top:0; right:0; width:100px; height:100px; opacity:0.1;">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100" width="100" height="100">
<path d="M50,10 C80,10 100,30 100,50 C100,80 80,100 50,100 C20,100 0,80 0,50 C0,30 20,10 50,10 Z" fill="white"></path>
<path d="M50,30 C66.57,30 80,43.43 80,60 C80,76.57 66.57,90 50,90 C33.43,90 20,76.57 20,60 C20,43.43 33.43,30 50,30 Z" fill="none" stroke="white" stroke-width="5"></path>
<circle cx="50" cy="60" r="10" fill="white"></circle>
</svg>
</div>
<div style="position:relative; z-index:1;">
<div style="display:flex; align-items:center; margin-bottom:15px;">
<div style="width:50px; height:50px; border-radius:50%; background-color:rgba(255,255,255,0.2); display:flex; align-items:center; justify-content:center; margin-right:15px; animation:pulse 2s infinite;">
<span style="font-size:28px;">🚨</span>
</div>
<div>
<h2 style="margin:0; font-weight:500; font-size:22px;">[触发中] {{ .CommonLabels.alertname }}</h2>
<p style="margin:0; font-size:14px; opacity:0.8;">紧急程度: {{ .CommonLabels.severity | title }}</p>
</div>
</div>
</div>
</div>
{{- else -}}
<div style="max-width:750px; margin:0 auto; font-family:'Inter',-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,'Helvetica Neue',Arial,sans-serif; background-color:#f9fafb; border-radius:12px; overflow:hidden; box-shadow:0 10px 25px -5px rgba(0,0,0,0.1),0 8px 10px -6px rgba(0,0,0,0.1);">
<div style="background:linear-gradient(135deg, #43a047 0%, #2e7d32 100%); color:white; padding:25px 30px; position:relative; overflow:hidden;">
<div style="position:absolute; top:0; right:0; width:100px; height:100px; opacity:0.1;">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100" width="100" height="100">
<path d="M50,10 C80,10 100,30 100,50 C100,80 80,100 50,100 C20,100 0,80 0,50 C0,30 20,10 50,10 Z" fill="white"></path>
<path d="M30,50 L50,70 L75,35" fill="none" stroke="white" stroke-width="8" stroke-linecap="round"></path>
</svg>
</div>
<div style="position:relative; z-index:1;">
<div style="display:flex; align-items:center; margin-bottom:15px;">
<div style="width:50px; height:50px; border-radius:50%; background-color:rgba(255,255,255,0.2); display:flex; align-items:center; justify-content:center; margin-right:15px;">
<span style="font-size:28px;">✅</span>
</div>
<div>
<h2 style="margin:0; font-weight:500; font-size:22px;">[已解决] {{ .CommonLabels.alertname }}</h2>
<p style="margin:0; font-size:14px; opacity:0.8;">状态已恢复正常</p>
</div>
</div>
</div>
</div>
{{- end }}

<div style="padding:30px; background-color:white;">
{{- if .Alerts -}}
{{- range .Alerts }}
<div style="margin-bottom:25px; border-radius:10px; overflow:hidden; box-shadow:0 4px 6px -1px rgba(0,0,0,0.1),0 2px 4px -1px rgba(0,0,0,0.06); transition:transform 0.3s ease, box-shadow 0.3s ease;">
<div style="padding:20px; background-color:
{{– if eq $.Status "firing" -}}
{{- if eq .Labels.severity "critical" }}#ffebee{{ else if eq .Labels.severity "warning" }}#fff3e0{{ else }}#e8f5e9{{ end -}}
{{- else -}}
#e8f5e9
{{- end -}};
border-left:4px solid;
border-left-color:
{{- if eq $.Status "firing" -}}
{{- if eq .Labels.severity "critical" }}#e53935{{ else if eq .Labels.severity "warning" }}#fb8c00{{ else }}#43a047{{ end -}}
{{- else -}}
#43a047
{{- end -}};">

<div style="display:flex; justify-content:space-between; align-items:flex-start;">
<div style="display:flex; align-items:center;">
<div style="width:40px; height:40px; border-radius:50%; background-color:
{{– if eq $.Status "firing" -}}
{{- if eq .Labels.severity "critical" }}rgba(229,57,53,0.1){{ else if eq .Labels.severity "warning" }}rgba(251,140,0,0.1){{ else }}rgba(67,160,71,0.1){{ end -}}
{{- else -}}
rgba(67,160,71,0.1)
{{- end -}};
display:flex; align-items:center; justify-content:center; margin-right:15px;">
<span style="font-size:20px; color:
{{– if eq $.Status "firing" -}}
{{- if eq .Labels.severity "critical" }}#e53935{{ else if eq .Labels.severity "warning" }}#fb8c00{{ else }}#43a047{{ end -}}
{{- else -}}
#43a047
{{- end -}};">
{{- if eq $.Status "firing" -}}
{{- if eq .Labels.severity "critical" }}🔥{{ else if eq .Labels.severity "warning" }}⚠️{{ else }}🔍{{ end -}}
{{- else -}}
✅
{{- end -}}
</span>
</div>
<h3 style="margin:0; color:#2d3748; font-weight:500; font-size:18px;">{{ .Labels.alertname }}</h3>
</div>
<div style="padding:4px 12px; border-radius:4px; background-color:
{{– if eq $.Status "firing" -}}
{{- if eq .Labels.severity "critical" }}rgba(229,57,53,0.1){{ else if eq .Labels.severity "warning" }}rgba(251,140,0,0.1){{ else }}rgba(67,160,71,0.1){{ end -}}
{{- else -}}
rgba(67,160,71,0.1)
{{- end -}};
color:
{{- if eq $.Status "firing" -}}
{{- if eq .Labels.severity "critical" }}#e53935{{ else if eq .Labels.severity "warning" }}#fb8c00{{ else }}#43a047{{ end -}}
{{- else -}}
#43a047
{{- end -}};
font-size:13px; font-weight:500;">
{{- if eq $.Status "firing" -}}
{{ .Labels.severity | title }}
{{- else -}}
已解决
{{- end -}}
</div>
</div>

<div style="margin-top:15px; padding-left:55px;">
<div style="display:grid; grid-template-columns:1fr 1fr; gap:10px;">
<div style="padding:10px; background-color:rgba(0,0,0,0.03); border-radius:6px;">
<p style="margin:0; font-size:13px; color:#718096;">实例</p>
<p style="margin:0; font-size:16px; font-weight:500;">{{ .Labels.instance }}</p>
</div>
<div style="padding:10px; background-color:rgba(0,0,0,0.03); border-radius:6px;">
<p style="margin:0; font-size:13px; color:#718096;">触发时间</p>
<p style="margin:0; font-size:16px; font-weight:500;">{{ .StartsAt.Format "2006-01-02 15:04:05" }}</p>
</div>
</div>

<div style="margin-top:15px; padding:15px; background-color:rgba(0,0,0,0.03); border-radius:6px;">
<p style="margin:0 0 5px 0; font-size:13px; color:#718096;">描述</p>
<p style="margin:0; font-size:16px;">{{ .Annotations.summary }}</p>
</div>

{{- if .Annotations.description -}}
<div style="margin-top:15px; padding:15px; background-color:rgba(0,0,0,0.03); border-radius:6px;">
<p style="margin:0 0 5px 0; font-size:13px; color:#718096;">详细信息</p>
<p style="margin:0; font-size:16px;">{{ .Annotations.description }}</p>
</div>
{{- end -}}

{{- if .Annotations.runbook_url -}}
<div style="margin-top:15px;">
<a href="{{ .Annotations.runbook_url }}" style="display:inline-flex; align-items:center; padding:8px 16px; background-color:#1976d2; color:white; border-radius:6px; text-decoration:none; font-weight:500; transition:background-color 0.3s ease;">
<span style="margin-right:8px;">📖</span>
查看操作手册
</a>
</div>
{{- end -}}
</div>
</div>
</div>
{{- end -}}
{{- else -}}
<div style="text-align:center; padding:40px 0; color:#6b7280;">
<div style="font-size:64px; color:#e5e7eb; margin-bottom:20px;">✅</div>
<h3 style="margin:0 0 10px 0; font-weight:400; color:#374151;">一切正常</h3>
<p style="margin:0; max-width:300px; margin:0 auto;">没有活跃的告警需要处理</p>
</div>
{{- end -}}
</div>

<div style="padding:15px 30px; background-color:#f9fafb; border-top:1px solid #e5e7eb; text-align:center; font-size:13px; color:#9ca3af;">
<p style="margin:0;">此告警由 Alertmanager 自动生成</p>
<p style="margin:5px 0 0 0;">查看所有告警: <a href="http://alertmanager-server:9093" style="color:#1976d2; text-decoration:none;">http://alertmanager-server:9093</a></p>
</div>
</div>

<style>
@keyframes pulse {
0% { transform: scale(1); box-shadow: 0 0 0 0 rgba(239, 83, 80, 0.7); }
70% { transform: scale(1.05); box-shadow: 0 0 0 10px rgba(239, 83, 80, 0); }
100% { transform: scale(1); box-shadow: 0 0 0 0 rgba(239, 83, 80, 0); }
}
</style>
{{ end }}

#启动alertmanager服务
systemctl start alertmanager.service

**方案二：**钉钉配置

vim alertmanager.yml

global:
# 移除 SMTP 相关配置（如果不再使用邮箱）

templates:
– '/usr/local/alertmanager-0.28.1/custom.tmpl' #钉钉模板

route:
receiver: 'dingtalk-notification' # 默认接收器改为钉钉
group_wait: 0s
group_interval: 30s
repeat_interval: 300s

receivers:
– name: 'dingtalk-notification'
webhook_configs:
– url: 'https://oapi.dingtalk.com/robot/send?access_token' #钉钉的Webhook
send_resolved: true
http_config:
tls_config:
insecure_skip_verify: true
headers:
Content-Type: 'application/json'
body: '{{ template "dingtalk_alert_template" . }}' # 引用钉钉模板

vim custom.tmpl

{{/* 钉钉模板（JSON Markdown 格式） */}}
{{ define "dingtalk_alert_template" }}
{
"msgtype": "markdown",
"markdown": {
"title": "[{{ if eq .Status \\"firing\\" }}🔥告警触发{{ else }}✅告警恢复{{ end }}] {{ .CommonLabels.alertname }}",
"text": "### {{ if eq .Status \\"firing\\" }}🔥告警触发（级别：{{ .CommonLabels.severity | toUpper }}）{{ else }}✅告警恢复{{ end }}\\n" +
"**告警名称**: {{ .CommonLabels.alertname }}\\n" +
"**实例**: {{ .CommonLabels.instance }}\\n" +
"**描述**: {{ .CommonAnnotations.description }}\\n\\n" +
"{{ range .Alerts }}\\n" +
"- **详情**: {{ .Annotations.summary }}\\n" +
"- **时间**: {{ .StartsAt.Format \\"2006-01-02 15:04:05\\" }}\\n" +
"{{ end }}"
}
}
{{ end }}

两者可以同时使用，文档这里不做配置

7.3 触发报警规则配置

#创建规则文件
vim /usr/local/prometheus-2.29.1/alert-rules.yml

groups:
# 主机资源监控规则（基础架构稳定性）
– name: host–resource–rules
rules:
# 节点 CPU 使用率过高（系统负载）
– alert: 主机CPU使用率过高
expr: (100 – avg by(instance)(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
for: 20s
labels:
severity: 严重
annotations:
summary: "主机 {{ $labels.instance }} CPU 使用率过高 (当前值: {{ $value | printf \\"%.2f\\" }}%)"
description: "主机 CPU 使用率持续超过 80%，可能导致服务响应缓慢，需检查高负载进程。"

# 节点内存使用率过高（防止 OOM）
– alert: 主机内存使用率过高
expr: (node_memory_MemTotal_bytes – node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 85
for: 20s
labels:
severity: 严重
annotations:
summary: "主机 {{ $labels.instance }} 内存使用率过高 (当前值: {{ $value | printf \\"%.2f\\" }}%)"
description: "主机内存使用率持续超过 85%，可能触发 OOM Killer，导致服务异常终止。"

# 节点磁盘空间不足（避免系统崩溃）
– alert: 磁盘空间不足
expr: (1 – node_filesystem_free_bytes{fstype=~"ext4|xfs|btrfs"} / node_filesystem_size_bytes{fstype=~"ext4|xfs|btrfs"}) * 100 > 80
for: 20s
labels:
severity: 警告
annotations:
summary: "主机 {{ $labels.instance }} 磁盘分区 {{ $labels.mountpoint }} 空间不足 (当前值: {{ $value | printf \\"%.2f\\" }}%)"
description: "磁盘空间使用率超过 80%，可能导致日志写入失败、应用无法启动，需清理或扩容。"

# 节点磁盘 IO 延迟过高（影响读写性能）
– alert: 磁盘IO延迟过高
expr: avg by(instance, device)(irate(node_disk_io_time_seconds_total[5m])) > 100
for: 20s
labels:
severity: 警告
annotations:
summary: "主机 {{ $labels.instance }} 磁盘 {{ $labels.device }} IO 延迟过高 (当前值: {{ $value | printf \\"%.2f\\" }}ms)"
description: "磁盘 IO 延迟持续超过 100ms，可能由磁盘故障、碎片过多或负载过高导致。"

# 节点网络带宽利用率过高（网络瓶颈）
– alert: 网络带宽利用率过高
expr: sum by(instance, device)(rate(node_network_receive_bytes_total[5m]) + rate(node_network_transmit_bytes_total[5m])) / node_network_speed_bytes * 100 > 80
for: 20s
labels:
severity: 警告
annotations:
summary: "主机 {{ $labels.instance }} 网络接口 {{ $labels.device }} 带宽利用率过高 (当前值: {{ $value | printf \\"%.2f\\" }}%)"
description: "网络带宽利用率持续超过 80%，可能导致数据包丢失、服务响应超时。"

# 节点负载过高（系统整体压力）
– alert: 系统负载过高
expr: node_load1 > on (instance) node_cpu_seconds_total{mode="idle"} * 0.8
for: 20s
labels:
severity: 警告
annotations:
summary: "主机 {{ $labels.instance }} 系统负载过高 (当前值: {{ $value | printf \\"%.2f\\" }})"
description: "系统 1 分钟负载超过 CPU 核心数的 80%，可能由进程泄漏或资源竞争导致。"

# 节点不可达（基础设施可用性）
– alert: 节点不可达
expr: up{job="node–exporter"} == 0
for: 20s
labels:
severity: 严重
annotations:
summary: "主机 {{ $labels.instance }} 不可达"
description: "Node Exporter 无法访问，可能由主机崩溃、网络中断或服务异常导致。"

# Kubernetes 资源监控规则（容器平台稳定性）
– name: kubernetes–resource–rules
rules:
# Pod CPU 使用率过高（容器资源）
– alert: PodCPU使用率过高
expr: sum by(namespace, pod)(rate(container_cpu_usage_seconds_total{container!="POD", container!=""}[5m])) / sum by(namespace, pod)(container_spec_cpu_quota{container!="POD", container!=""}) * 100 > 80
for: 5m
labels:
severity: 警告
annotations:
summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} CPU 使用率过高 (当前值: {{ $value | printf \\"%.2f\\" }}%)"
description: "Pod CPU 使用率持续超过 80%，可能导致容器被限流（throttled），影响服务性能。"

# Pod 内存使用率过高（容器资源）
– alert: Pod内存使用率过高
expr: sum by(namespace, pod)(container_memory_usage_bytes{container!="POD", container!=""}) / sum by(namespace, pod)(container_spec_memory_limit_bytes{container!="POD", container!=""}) * 100 > 90
for: 5m
labels:
severity: 警告
annotations:
summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} 内存使用率过高 (当前值: {{ $value | printf \\"%.2f\\" }}%)"
description: "Pod 内存使用率持续超过 90%，可能触发 OOM Kill，导致容器重启。"

# Deployment 副本不足（服务可用性）
– alert: Deployment副本不足
expr: kube_deployment_spec_replicas != kube_deployment_status_available_replicas
for: 10m
labels:
severity: 严重
annotations:
summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} 副本不足"
description: "期望 {{ $value | printf \\"%.0f\\" }} 个副本，但只有 {{ $labels.available_replicas }} 个可用，可能影响服务可用性。"

# PVC 容量不足（存储资源）
– alert: PVCCapacity不足
expr: (1 – kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes) * 100 > 80
for: 24h
labels:
severity: 警告
annotations:
summary: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} 容量不足 (当前值: {{ $value | printf \\"%.2f\\" }}%)"
description: "PVC 容量使用率超过 80%，建议在 24 小时内扩容，避免影响数据写入。"

# 应用程序监控规则（直接关联业务）
– name: application–rules
rules:
# HTTP 请求错误率过高（业务可用性）
– alert: HTTP请求错误率过高
expr: sum by(job)(rate(http_requests_total{status=~"5.."}[5m])) / sum by(job)(rate(http_requests_total[5m])) * 100 > 5
for: 10m
labels:
severity: 警告
annotations:
summary: "服务 {{ $labels.job }} HTTP 请求错误率过高 (当前值: {{ $value | printf \\"%.2f\\" }}%)"
description: "HTTP 5xx 错误率持续超过 5%，可能由服务崩溃、依赖故障或资源不足导致。"

# 数据库连接池耗尽（数据访问）
– alert: 数据库连接池耗尽
expr: database_connections_active / database_connections_max * 100 > 90
for: 5m
labels:
severity: 严重
annotations:
summary: "数据库 {{ $labels.instance }} 连接池耗尽 (当前值: {{ $value | printf \\"%.2f\\" }}%)"
description: "数据库连接池使用率超过 90%，新请求可能无法获取连接，导致服务无响应。"

# 消息队列堆积（异步处理）
– alert: 消息队列堆积
expr: sum by(queue)(rabbitmq_queue_messages_ready) > 10000
for: 10m
labels:
severity: 警告
annotations:
summary: "消息队列 {{ $labels.queue }} 堆积严重 (当前值: {{ $value | printf \\"%.0f\\" }} 条)"
description: "消息队列中待处理消息超过 10000 条，消费者可能存在瓶颈，需检查消费逻辑或扩容。"

7.4 Prometheus连接配置

vim /usr/local/prometheus-2.29.1/prometheus.yml

alerting:
alertmanagers:
– static_configs:
– targets: ["192.168.200.14:9093"] #Alertmanager服务地址（默认端口9093）

rule_files:
– "/usr/local/prometheus-2.29.1/alert-rules.yml" #报警规则文件路径

#重启启动prometheus服务
systemctl restart prometheus.service

7.5 CPU触发报警测试

测试192.168.200.14主机cpu达到阈值触发报警

#在200.104编写测试脚本
vim cpu-test.sh

#!/usr/bin/env bash
# 文件名：cpu_stress.sh
# 用法：chmod +x cpu_stress.sh && ./cpu_stress.sh [持续时长秒]
# 默认持续 60 秒，如果传入参数则按参数秒数运行。

# 运行时长（秒）
DURATION=${1:-60}

# 计算结束时间
END_TIME=$((SECONDS + DURATION))

# 启动与 CPU 核数相同的子进程，每个子进程都在忙循环
CPU_CORES=$(nproc)
echo ">> 启动 $CPU_CORES 个 busy-loop，压满所有 CPU 核心，共运行 $DURATION 秒"

for ((i=1; i<=CPU_CORES; i++)); do
(
# 子进程忙循环直到超时
while [ $SECONDS -lt $END_TIME ]; do
: # 空操作，占用 CPU
done
) &
done

# 等待所有子进程结束
wait
echo ">> CPU 压力测试结束。"

#执行脚本（执行脚本前使用 journalctl -u alertmanager.service -f 查看日志）
journalctl -u alertmanager.service -f

bash cpu-test.sh

#日志
5月 14 03:01:39 prometheus alertmanager[7188]: time=2025-05-14T07:01:39.747Z level=DEBUG source=dispatch.go:165 msg="Received alert" component=dispatcher alert=主机CPU使用率过高[c57a700][resolved]
5月 14 03:01:54 prometheus alertmanager[7188]: time=2025-05-14T07:01:54.746Z level=DEBUG source=dispatch.go:165 msg="Received alert" component=dispatcher alert=主机CPU使用率过高[8ccbb6e][resolved]
5月 14 03:01:54 prometheus alertmanager[7188]: time=2025-05-14T07:01:54.752Z level=DEBUG source=dispatch.go:530 msg=flushing component=dispatcher aggrGroup={}:{} alerts="[主机CPU使用率过高[c57a700][resolved] 主机CPU使用率过高[8ccbb6e][resolved]]"
5月 14 03:01:56 prometheus alertmanager[7188]: time=2025-05-14T07:01:56.043Z level=DEBUG source=notify.go:876 msg="Notify success" component=dispatcher receiver=邮件接收人 integration=email[0] aggrGroup={}:{} attempts=1 duration=1.291165694s alerts="[主机CPU使用率过高[c57a700][resolved] 主机CPU使用率过高[8ccbb6e][resolved]]"

7.6 查看邮箱邮件