Kafka监控体系搭建：从JMX指标到Prometheus+Grafana告警
Kafka作为分布式消息队列系统，在生产环境中需要全方位的监控来确保系统稳定性和性能。完整的Kafka监控体系应涵盖集群健康状态、消息处理能力、存储使用情况以及网络吞吐量等关键指标。
基于JMX的Kafka监控架构主要包括三个层次：数据层（Kafka集群）、采集层（JMX指标采集）和展示层（Prometheus+Grafana）。通过这种架构，我们可以实时监控Kafka的运行状态，及时发现并解决问题。
Kafka暴露了丰富的JMX指标，这些指标可以帮助我们了解系统运行状况。关键指标包括：
- Broker级别指标：
- kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec
- kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
- kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec
- kafka.server:type=ReplicaManager,name=IsrShrinksPerSec
- kafka.server:type=ReplicaManager,name=IsrExpandsPerSec
- Topic级别指标：
- kafka.server:type=TopicMetrics,name=MessagesInPerSec
- kafka.server:type=TopicMetrics,name=BytesInPerSec
- kafka.server:type=TopicMetrics,name=BytesOutPerSec
- 消费者组指标：
- kafka.consumer:type=consumer-fetch-manager-metrics,name=records-consumed-rate
- kafka.consumer:type=consumer-fetch-manager-metrics,name=bytes-consumed-rate
配置JMX指标采集需要设置Kafka启动参数：
# 在server.properties中配置
# JMX监听端口
export JMX_PORT=9999
# 启用JMX
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=localhost"
要将Kafka JMX指标暴露给Prometheus，我们可以使用Jmx Prometheus HTTP Exporter。首先下载并运行exporter：
# 下载并运行JMX exporter
java -jar jmx_prometheus_httpserver.jar <port> config.yml
配置文件config.yml示例：
lowercaseOutputLabelNames: true
lowercaseOutputName: true
whitelistObjectNames:
– "kafka.*"
rules:
– pattern: "kafka.<type><name>\\\\.(?<counter>.+)\\\\.Count"
name: kafka_<type>_<name>_<counter>_count
type: COUNTER
– pattern: "kafka.<type><name>\\\\.(?<gauge>.+)\\\\.Value"
name: kafka_<type>_<name>_<gauge>_value
type: GAUGE
然后在Prometheus的配置文件中添加Kafka JMX目标：
scrape_configs:
– job_name: 'kafka_jmx'
static_configs:
– targets: ['localhost:9999']
Grafana可以用来可视化Prometheus收集的Kafka指标。首先需要添加Prometheus数据源，然后创建仪表板。
关键仪表板包括：
告警规则设计：
groups:
– name: kafka_alerts
rules:
– alert: KafkaBrokerHighDiskUsage
expr: kafka_log_log_size_bytes{topic!~".*internal.*"} / kafka_log_log_dirs_size_bytes > 0.85
for: 5m
labels:
severity: critical
annotations:
summary: "Kafka broker disk usage is high"
description: "Disk usage for {{ $labels.instance }} is {{ $value | printf "%.2f" }}%"
– alert: KafkaConsumerLagHigh
expr: sum(kafka_consumer_fetch_manager_records_consumed_total{topic=~"your_topic"})
– sum(kafka_consumer_fetch_manager_records_consumed_total{topic=~"your_topic", consumer_group="$consumer_group"}) > 10000
for: 2m
labels:
severity: warning
annotations:
summary: "Kafka consumer lag is high"
description: "Consumer lag for {{ $labels.consumer_group }} is {{ $value }} messages"
– alert: KafkaUnderReplicatedPartitions
expr: kafka_server_replicametrics_underreplicatedpartitions > 0
for: 5m
labels:
severity: critical
annotations:
summary: "Kafka has under replicated partitions"
description: "{{ $value }} partitions are under replicated"
下面是一个完整的Kafka监控部署示例：
# docker-compose.yml
version: '3'
services:
prometheus:
image: prom/prometheus:latest
ports:
– "9090:9090"
volumes:
– ./prometheus.yml:/etc/prometheus/prometheus.yml
– prometheus_data:/prometheus
command:
– '–config.file=/etc/prometheus/prometheus.yml'
– '–storage.tsdb.path=/prometheus'
– '–web.console.libraries=/etc/prometheus/console_libraries'
– '–web.console.templates=/etc/prometheus/consoles'
– '–storage.tsdb.retention.time=200h'
– '–web.enable-lifecycle'
grafana:
image: grafana/grafana:latest
ports:
– "3000:3000"
volumes:
– grafana_data:/var/lib/grafana
environment:
– GF_SECURITY_ADMIN_PASSWORD=admin
jmx-exporter:
image: prom/jmx-exporter:latest
ports:
– "5555:5555"
volumes:
– ./jmx-config.yml:/etc/config.yml
command:
– '–config.file=/etc/config.yml'
kafka:
image: confluentinc/cp-kafka:latest
ports:
– "9092:9092"
depends_on:
– zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_JMX_PORT: 9999
KAFKA_JMX_OPTS: -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
zookeeper:
image: confluentinc/cp-zookeeper:latest
ports:
– "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
volumes:
prometheus_data:
grafana_data:
最佳实践：
Kafka监控流程图
#publish-mermaid-1788282421526-0{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#publish-mermaid-1788282421526-0 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#publish-mermaid-1788282421526-0 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#publish-mermaid-1788282421526-0 .error-icon{fill:#552222;}#publish-mermaid-1788282421526-0 .error-text{fill:#552222;stroke:#552222;}#publish-mermaid-1788282421526-0 .edge-thickness-normal{stroke-width:1px;}#publish-mermaid-1788282421526-0 .edge-thickness-thick{stroke-width:3.5px;}#publish-mermaid-1788282421526-0 .edge-pattern-solid{stroke-dasharray:0;}#publish-mermaid-1788282421526-0 .edge-thickness-invisible{stroke-width:0;fill:none;}#publish-mermaid-1788282421526-0 .edge-pattern-dashed{stroke-dasharray:3;}#publish-mermaid-1788282421526-0 .edge-pattern-dotted{stroke-dasharray:2;}#publish-mermaid-1788282421526-0 .marker{fill:#333333;stroke:#333333;}#publish-mermaid-1788282421526-0 .marker.cross{stroke:#333333;}#publish-mermaid-1788282421526-0 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#publish-mermaid-1788282421526-0 p{margin:0;}#publish-mermaid-1788282421526-0 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#publish-mermaid-1788282421526-0 .cluster-label text{fill:#333;}#publish-mermaid-1788282421526-0 .cluster-label span{color:#333;}#publish-mermaid-1788282421526-0 .cluster-label span p{background-color:transparent;}#publish-mermaid-1788282421526-0 .label text,#publish-mermaid-1788282421526-0 span{fill:#333;color:#333;}#publish-mermaid-1788282421526-0 .node rect,#publish-mermaid-1788282421526-0 .node circle,#publish-mermaid-1788282421526-0 .node ellipse,#publish-mermaid-1788282421526-0 .node polygon,#publish-mermaid-1788282421526-0 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#publish-mermaid-1788282421526-0 .rough-node .label text,#publish-mermaid-1788282421526-0 .node .label text,#publish-mermaid-1788282421526-0 .image-shape .label,#publish-mermaid-1788282421526-0 .icon-shape .label{text-anchor:middle;}#publish-mermaid-1788282421526-0 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#publish-mermaid-1788282421526-0 .rough-node .label,#publish-mermaid-1788282421526-0 .node .label,#publish-mermaid-1788282421526-0 .image-shape .label,#publish-mermaid-1788282421526-0 .icon-shape .label{text-align:center;}#publish-mermaid-1788282421526-0 .node.clickable{cursor:pointer;}#publish-mermaid-1788282421526-0 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#publish-mermaid-1788282421526-0 .arrowheadPath{fill:#333333;}#publish-mermaid-1788282421526-0 .edgePath .path{stroke:#333333;stroke-width:1px;}#publish-mermaid-1788282421526-0 .flowchart-link{stroke:#333333;fill:none;}#publish-mermaid-1788282421526-0 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#publish-mermaid-1788282421526-0 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#publish-mermaid-1788282421526-0 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#publish-mermaid-1788282421526-0 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#publish-mermaid-1788282421526-0 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#publish-mermaid-1788282421526-0 .cluster text{fill:#333;}#publish-mermaid-1788282421526-0 .cluster span{color:#333;}#publish-mermaid-1788282421526-0 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#publish-mermaid-1788282421526-0 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#publish-mermaid-1788282421526-0 rect.text{fill:none;stroke-width:0;}#publish-mermaid-1788282421526-0 .icon-shape,#publish-mermaid-1788282421526-0 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#publish-mermaid-1788282421526-0 .icon-shape p,#publish-mermaid-1788282421526-0 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#publish-mermaid-1788282421526-0 .icon-shape .label rect,#publish-mermaid-1788282421526-0 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#publish-mermaid-1788282421526-0 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#publish-mermaid-1788282421526-0 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#publish-mermaid-1788282421526-0 .node .neo-node{stroke:#9370DB;}#publish-mermaid-1788282421526-0 [data-look=\”neo\”].node rect,#publish-mermaid-1788282421526-0 [data-look=\”neo\”].cluster rect,#publish-mermaid-1788282421526-0 [data-look=\”neo\”].node polygon{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788282421526-0 [data-look=\”neo\”].swimlane.cluster rect{filter:none;}#publish-mermaid-1788282421526-0 [data-look=\”neo\”].node path{stroke:#9370DB;stroke-width:1px;}#publish-mermaid-1788282421526-0 [data-look=\”neo\”].node .outer-path{filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788282421526-0 [data-look=\”neo\”].node .neo-line path{stroke:#9370DB;filter:none;}#publish-mermaid-1788282421526-0 [data-look=\”neo\”].node circle{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788282421526-0 [data-look=\”neo\”].node circle .state-start{fill:#000000;}#publish-mermaid-1788282421526-0 [data-look=\”neo\”].icon-shape .icon{fill:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788282421526-0 [data-look=\”neo\”].icon-shape .icon-neo path{stroke:#9370DB;filter:drop-shadow(1px 2px 2px rgba(185, 185, 185, 1));}#publish-mermaid-1788282421526-0 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}JMX指标HTTP请求数据存储查询可视化展示告警规则短信/邮件
Kafka集群
JMX Exporter
Prometheus
Grafana
监控仪表板
告警通知
运维团队
Kafka关键指标监控表
| 指标类型 | 指标名称 | 含义 | 告警阈值 | 处理方式 |
|———|———|——|———|———|
| Broker | BytesInPerSec | Broker每秒接收字节数 | > 100MB/s | 检查生产者是否有突发流量 |
| Broker | BytesOutPerSec | Broker每秒发送字节数 | > 100MB/s | 检查消费者处理能力 |
| Broker | UnderReplicatedPartitions | 副本不足的分区数 | > 0 | 检查副本同步状态 |
| Topic | MessagesInPerSec | 每秒消息数 | > 50000/s | 检查Topic配置 |
| Topic | AverageLogTime | 日志平均延迟 | > 5s | 检查消费者消费情况 |
| Consumer | ConsumerLag | 消费延迟 | > 10000条 | 检查消费者处理能力 |
| Consumer | RecordsConsumedRate | 消费速率 | < 1000/s | 检查消费者状态 |
最小运行示例与注意事项
注意事项：
