基于 Prometheus 构建 AI 模型监控体系：从指标采集到智能告警

基于 Prometheus 构建 AI 模型监控体系：从指标采集到智能告警

前言

上周半夜，运维同学小张突然给我打电话：“涵姐，线上的推理服务响应时间突破 10 秒了！”

我赶紧打开监控面板，发现 GPU 利用率只有 20%，但内存占用却高达 95%。原来是模型加载时的内存泄漏导致服务性能急剧下降。

"要是能早点发现这个趋势就好了。"小张感慨道。

确实，AI 模型的监控和传统服务有很大不同。除了常规的 CPU、内存指标，我们还需要关注推理延迟、吞吐量、模型精度等业务指标。

今天我就带大家一起，构建一套完整的 AI 模型监控体系。

一、底层原理

1.1 AI 监控指标体系

#mermaid-svg-4Q3r7YjSX1DUN08n{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-4Q3r7YjSX1DUN08n .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-4Q3r7YjSX1DUN08n .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-4Q3r7YjSX1DUN08n .error-icon{fill:#552222;}#mermaid-svg-4Q3r7YjSX1DUN08n .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-4Q3r7YjSX1DUN08n .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-4Q3r7YjSX1DUN08n .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-4Q3r7YjSX1DUN08n .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-4Q3r7YjSX1DUN08n .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-4Q3r7YjSX1DUN08n .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-4Q3r7YjSX1DUN08n .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-4Q3r7YjSX1DUN08n .marker{fill:#333333;stroke:#333333;}#mermaid-svg-4Q3r7YjSX1DUN08n .marker.cross{stroke:#333333;}#mermaid-svg-4Q3r7YjSX1DUN08n svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-4Q3r7YjSX1DUN08n p{margin:0;}#mermaid-svg-4Q3r7YjSX1DUN08n .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-4Q3r7YjSX1DUN08n .cluster-label text{fill:#333;}#mermaid-svg-4Q3r7YjSX1DUN08n .cluster-label span{color:#333;}#mermaid-svg-4Q3r7YjSX1DUN08n .cluster-label span p{background-color:transparent;}#mermaid-svg-4Q3r7YjSX1DUN08n .label text,#mermaid-svg-4Q3r7YjSX1DUN08n span{fill:#333;color:#333;}#mermaid-svg-4Q3r7YjSX1DUN08n .node rect,#mermaid-svg-4Q3r7YjSX1DUN08n .node circle,#mermaid-svg-4Q3r7YjSX1DUN08n .node ellipse,#mermaid-svg-4Q3r7YjSX1DUN08n .node polygon,#mermaid-svg-4Q3r7YjSX1DUN08n .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-4Q3r7YjSX1DUN08n .rough-node .label text,#mermaid-svg-4Q3r7YjSX1DUN08n .node .label text,#mermaid-svg-4Q3r7YjSX1DUN08n .image-shape .label,#mermaid-svg-4Q3r7YjSX1DUN08n .icon-shape .label{text-anchor:middle;}#mermaid-svg-4Q3r7YjSX1DUN08n .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-4Q3r7YjSX1DUN08n .rough-node .label,#mermaid-svg-4Q3r7YjSX1DUN08n .node .label,#mermaid-svg-4Q3r7YjSX1DUN08n .image-shape .label,#mermaid-svg-4Q3r7YjSX1DUN08n .icon-shape .label{text-align:center;}#mermaid-svg-4Q3r7YjSX1DUN08n .node.clickable{cursor:pointer;}#mermaid-svg-4Q3r7YjSX1DUN08n .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-4Q3r7YjSX1DUN08n .arrowheadPath{fill:#333333;}#mermaid-svg-4Q3r7YjSX1DUN08n .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-4Q3r7YjSX1DUN08n .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-4Q3r7YjSX1DUN08n .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-4Q3r7YjSX1DUN08n .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-4Q3r7YjSX1DUN08n .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-4Q3r7YjSX1DUN08n .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-4Q3r7YjSX1DUN08n .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-4Q3r7YjSX1DUN08n .cluster text{fill:#333;}#mermaid-svg-4Q3r7YjSX1DUN08n .cluster span{color:#333;}#mermaid-svg-4Q3r7YjSX1DUN08n div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-4Q3r7YjSX1DUN08n .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-4Q3r7YjSX1DUN08n rect.text{fill:none;stroke-width:0;}#mermaid-svg-4Q3r7YjSX1DUN08n .icon-shape,#mermaid-svg-4Q3r7YjSX1DUN08n .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-4Q3r7YjSX1DUN08n .icon-shape p,#mermaid-svg-4Q3r7YjSX1DUN08n .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-4Q3r7YjSX1DUN08n .icon-shape .label rect,#mermaid-svg-4Q3r7YjSX1DUN08n .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-4Q3r7YjSX1DUN08n .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-4Q3r7YjSX1DUN08n .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-4Q3r7YjSX1DUN08n :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

监控指标采集

基础设施层

服务层

模型层

CPU 利用率

GPU 利用率

内存使用

网络流量

请求延迟

请求吞吐量

错误率

并发数

推理延迟

模型准确率

漂移检测

置信度分布

1.2 指标类型对比

二、快速上手

2.1 环境准备

# 安装 Prometheus Operator
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/bundle.yaml

# 安装 Grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana -n monitoring

2.2 配置指标采集

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: llm–inference–monitor
namespace: monitoring
spec:
selector:
matchLabels:
app: llm–service
endpoints:
– port: http
path: /metrics
interval: 15s
scrapeTimeout: 10s

三、核心 API 与深水区

3.1 自定义模型指标暴露

from prometheus_client import start_http_server, Summary, Histogram, Gauge

# 定义指标
REQUEST_LATENCY = Histogram(
'llm_inference_latency_seconds',
'LLM inference latency',
['model', 'endpoint']
)

REQUEST_COUNT = Summary(
'llm_requests_total',
'Total LLM requests',
['model', 'status']
)

CONFIDENCE_SCORE = Gauge(
'llm_confidence_score',
'Model prediction confidence',
['model']
)

@REQUEST_LATENCY.time(model='llama3', endpoint='/inference')
def inference(prompt):
REQUEST_COUNT.labels(model='llama3', status='success').observe(1)

result = model.generate(prompt)
CONFIDENCE_SCORE.labels(model='llama3').set(result.confidence)

return result

if __name__ == '__main__':
start_http_server(8000)
# 启动推理服务…

3.2 配置告警规则

groups:
– name: llm–alerts
rules:
– alert: LLMHighLatency
expr: llm_inference_latency_seconds_p95 > 5
for: 5m
labels:
severity: critical
annotations:
summary: "LLM 推理延迟过高"
description: "P95 延迟超过 5 秒，当前值: {{ $value }}s"

– alert: LLMConfidenceDrop
expr: llm_confidence_score < 0.7
for: 10m
labels:
severity: warning
annotations:
summary: "模型置信度下降"
description: "模型置信度低于 70%，当前值: {{ $value }}"

四、实战演练

4.1 完整监控仪表盘配置

{
"title": "LLM 推理服务监控",
"panels": [
{
"type": "graph",
"title": "推理延迟趋势",
"targets": [
{"expr": "llm_inference_latency_seconds_p95", "legendFormat": "P95"}
]
},
{
"type": "gauge",
"title": "GPU 利用率",
"targets": [{"expr": "nvidia_gpu_utilization"}]
},
{
"type": "stat",
"title": "请求吞吐量",
"targets": [{"expr": "sum(rate(llm_requests_total[5m]))"}]
}
]
}

五、避坑指南

5.1 指标爆炸问题

⚠️ 存储压力：过多的标签维度导致指标数量呈指数级增长。

✅ 解决方案：限制标签维度，使用聚合规则减少存储开销。

5.2 忽略模型漂移检测

⚠️ 质量下降：模型在生产环境中性能逐渐下降但未被发现。

✅ 解决方案：定期检测数据分布变化和模型性能指标。

六、总结

构建 AI 监控体系就像是为你的模型服务装上了一双"千里眼"。它能帮你实时洞察系统状态，及时发现潜在问题。

希望今天的分享能帮助你构建更可靠的 AI 服务监控体系。如果你有任何问题或建议，欢迎在评论区与我交流。

好了，我得去给 Ping 准备它最喜欢的小鱼干了。下次再见！