提示：本文原创作品，良心制作，干货为主，简洁清晰，一看就会
钉钉告警
- 1.1 钉钉添加机器人
- 1.2 安装webhook
- 1.3 配置alertmanager-alertmanager.yaml
- 1.4 创建alertmanagerConfig
- 1.5 测试告警
1.1 钉钉添加机器人
在钉钉群设置中新增自定义机器人，填写机器人名称，安全校验优先选择加签模式，规避恶意调用导致的消息刷屏风险。创建完成后保存专属 Webhook 地址与加密密钥，这两组参数是后续告警配置的核心凭证
1.2 安装webhook
原生 Alertmanager 无法直接对接钉钉消息格式，需要部署钉钉 Webhook 转发插件实现报文格式转换 插件会接收 Alertmanager 推送的原生告警 JSON 数据，自动完成钉钉签名加密、报文格式封装，转换为钉钉机器人可识别的消息结构
root@k8s–master1:~# git clone https://github.com/timonwong/prometheus-webhook-dingtalk.git
root@k8s–master1:~# cd prometheus-webhook-dingtalk/contrib/k8s/
root@k8s–master1:~/prometheus–webhook–dingtalk/contrib/k8s# vim config/config.yaml
root@k8s–master1:~/prometheus–webhook–dingtalk/contrib/k8s# vim deployment.yaml
root@k8s–master1:~/prometheus–webhook–dingtalk/contrib/k8s# kubectl kustomize | kubectl apply -f – -n monitoring
root@k8s–master1:~/prometheus–webhook–dingtalk/contrib/k8s# kubectl get pod -n monitoring | grep ding
alertmanager–webhook–dingtalk–cb7f6c584–92sqj 1/1 Running 0 13s
1.3 配置alertmanager-alertmanager.yaml
root@k8s–master1:~/prometheus–webhook–dingtalk/contrib/k8s# cd /root/kube-prometheus/manifests/
root@k8s–master1:~/kube–prometheus/manifests# vim alertmanager-alertmanager.yaml
1.4 创建alertmanagerConfig
root@k8s–master1:~/kube–prometheus/manifests# vim dingding-alertmanagerconfig.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: dingding
labels:
# 需要和alertmanager-alertmanager.yaml中的告警配置标签保持一致
alertmanagerConfig: email
namespace: monitoring
spec:
route:
groupBy: ['severity']
groupWait: 1m
groupInterval: 1m
repeatInterval: 1m
receiver: dingding–webhook
receivers:
– name: "dingding-webhook"
webhookConfigs:
# 告警恢复时发送恢复通知
– sendResolved: true
# 钉钉告警webhook服务的访问地址
url: "http://alertmanager-webhook-dingtalk.monitoring/dingtalk/webhook1/send"
1.5 测试告警
我目前有一套mysql高可用集群，接下来将以mysql集群为例演示如何配置对应的 Prometheus 告警触发规则
root@k8s–master1:~/kube–prometheus/manifests# kubectl get pod
NAME READY STATUS RESTARTS AGE
mysql–rep–master–0 2/2 Running 8 (165m ago) 5d21h
mysql–rep–slave–0 2/2 Running 6 (165m ago) 5d21h
mysql–rep–slave–1 2/2 Running 6 (165m ago) 5d21h
## mysql告警规则
root@k8s–master1:~/kube–prometheus/manifests# vim mysql-rule.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
app: kube–prometheus–stack
role: alerting–rules
prometheus: kube–prometheus–stack–prometheus
name: prometheus–mysql–alerts
namespace: monitoring # 请替换为你的Prometheus所在namespace
spec:
groups:
– name: mysql
rules:
# ==================== 1. 集群可用性告警 ====================
– alert: MySQLDown
expr: mysql_up == 0
for: 1m
labels:
severity: critical
namespace: monitoring
annotations:
summary: "MySQL实例 {{ $labels.instance }} 已宕机"
description: "Prometheus 无法连接到 {{ $labels.pod }} 上的 MySQL 实例。这通常意味着 mysqld 进程已停止或 exporter 无法连接。"
# ==================== 2. 主从复制告警 ====================
# 2.1 复制延迟过高
– alert: MySQLReplicationLagHigh
expr: mysql_slave_status_seconds_behind_source > 30
for: 2m
labels:
severity: warning
namespace: monitoring
annotations:
summary: "MySQL 复制延迟较高"
description: "从库 {{ $labels.pod }} (实例: {{ $labels.instance }}) 复制落后主库 {{ $value }} 秒。请检查网络延迟或主库写入负载。"
# 2.2 复制线程停止
– alert: MySQLReplicationSQLThreadDown
expr: mysql_slave_status_replica_sql_running == 0
for: 1m
labels:
severity: critical
namespace: monitoring
annotations:
summary: "MySQL 复制 SQL 线程停止"
description: "从库 {{ $labels.pod }} 的 SQL 线程未运行，数据同步已中断。请检查 relay log 是否有损坏或错误。"
– alert: MySQLReplicationIOThreadDown
expr: mysql_slave_status_replica_io_running == 0
for: 1m
labels:
severity: critical
namespace: monitoring
annotations:
summary: "MySQL 复制 IO 线程停止"
description: "从库 {{ $labels.pod }} 的 IO 线程未运行，无法从主库获取二进制日志，网络连接可能已断开。"
进入mysql slave pod内部，关闭主从同步，测试一下能不能收到告警
root@k8s–master1:~/kube–prometheus/manifests# kubectl exec -it mysql-rep-slave-0 /bin/bash
I have no name!@mysql-rep-slave-0:/$ mysql –uroot –p'Root@12345'
mysql> stop replica;
收到告警消息 
进入mysql slave pod内部，恢复主从同步，测试一下能不能收到恢复消息
root@k8s–master1:~/kube–prometheus/manifests# kubectl exec -it mysql-rep-slave-0 /bin/bash
I have no name!@mysql-rep-slave-0:/$ mysql –uroot –p'Root@12345'
mysql> start replica;
收到恢复消息 
到此，Prometheus钉钉告警就到此结束了！
注： 文中若有疏漏，欢迎大家指正赐教。 本文为100%原创，转载请务必标注原创作者，尊重劳动成果。 求赞、求关注、求评论！你的支持是我更新的最大动力，评论区等你～
