监控与运维(基于 Elasticsearch 8.x)
学习目标
- 深入掌握 Elasticsearch 8.x 集群监控的所有方法和工具
- 理解所有关键性能指标和告警阈值
- 掌握完整的备份恢复策略和最佳实践
- 能够处理所有常见运维问题和故障排查
- 掌握自动化运维和容量规划
- 理解生产环境的高可用架构设计
知识结构思维导图
#mermaid-svg-yi79e0nj9tmwkUqY{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-yi79e0nj9tmwkUqY .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-yi79e0nj9tmwkUqY .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-yi79e0nj9tmwkUqY .error-icon{fill:#552222;}#mermaid-svg-yi79e0nj9tmwkUqY .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-yi79e0nj9tmwkUqY .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-yi79e0nj9tmwkUqY .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-yi79e0nj9tmwkUqY .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-yi79e0nj9tmwkUqY .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-yi79e0nj9tmwkUqY .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-yi79e0nj9tmwkUqY .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-yi79e0nj9tmwkUqY .marker{fill:#333333;stroke:#333333;}#mermaid-svg-yi79e0nj9tmwkUqY .marker.cross{stroke:#333333;}#mermaid-svg-yi79e0nj9tmwkUqY svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-yi79e0nj9tmwkUqY p{margin:0;}#mermaid-svg-yi79e0nj9tmwkUqY .edge{stroke-width:3;}#mermaid-svg-yi79e0nj9tmwkUqY .section–1 rect,#mermaid-svg-yi79e0nj9tmwkUqY .section–1 path,#mermaid-svg-yi79e0nj9tmwkUqY .section–1 circle,#mermaid-svg-yi79e0nj9tmwkUqY .section–1 polygon,#mermaid-svg-yi79e0nj9tmwkUqY .section–1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .section–1 text{fill:#ffffff;}#mermaid-svg-yi79e0nj9tmwkUqY .node-icon–1{font-size:40px;color:#ffffff;}#mermaid-svg-yi79e0nj9tmwkUqY .section-edge–1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .edge-depth–1{stroke-width:17;}#mermaid-svg-yi79e0nj9tmwkUqY .section–1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled,#mermaid-svg-yi79e0nj9tmwkUqY .disabled circle,#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:lightgray;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:#efefef;}#mermaid-svg-yi79e0nj9tmwkUqY .section-0 rect,#mermaid-svg-yi79e0nj9tmwkUqY .section-0 path,#mermaid-svg-yi79e0nj9tmwkUqY .section-0 circle,#mermaid-svg-yi79e0nj9tmwkUqY .section-0 polygon,#mermaid-svg-yi79e0nj9tmwkUqY .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-yi79e0nj9tmwkUqY .section-0 text{fill:black;}#mermaid-svg-yi79e0nj9tmwkUqY .node-icon-0{font-size:40px;color:black;}#mermaid-svg-yi79e0nj9tmwkUqY .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-yi79e0nj9tmwkUqY .edge-depth-0{stroke-width:14;}#mermaid-svg-yi79e0nj9tmwkUqY .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled,#mermaid-svg-yi79e0nj9tmwkUqY .disabled circle,#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:lightgray;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:#efefef;}#mermaid-svg-yi79e0nj9tmwkUqY .section-1 rect,#mermaid-svg-yi79e0nj9tmwkUqY .section-1 path,#mermaid-svg-yi79e0nj9tmwkUqY .section-1 circle,#mermaid-svg-yi79e0nj9tmwkUqY .section-1 polygon,#mermaid-svg-yi79e0nj9tmwkUqY .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .section-1 text{fill:black;}#mermaid-svg-yi79e0nj9tmwkUqY .node-icon-1{font-size:40px;color:black;}#mermaid-svg-yi79e0nj9tmwkUqY .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .edge-depth-1{stroke-width:11;}#mermaid-svg-yi79e0nj9tmwkUqY .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled,#mermaid-svg-yi79e0nj9tmwkUqY .disabled circle,#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:lightgray;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:#efefef;}#mermaid-svg-yi79e0nj9tmwkUqY .section-2 rect,#mermaid-svg-yi79e0nj9tmwkUqY .section-2 path,#mermaid-svg-yi79e0nj9tmwkUqY .section-2 circle,#mermaid-svg-yi79e0nj9tmwkUqY .section-2 polygon,#mermaid-svg-yi79e0nj9tmwkUqY .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .section-2 text{fill:#ffffff;}#mermaid-svg-yi79e0nj9tmwkUqY .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-yi79e0nj9tmwkUqY .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .edge-depth-2{stroke-width:8;}#mermaid-svg-yi79e0nj9tmwkUqY .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled,#mermaid-svg-yi79e0nj9tmwkUqY .disabled circle,#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:lightgray;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:#efefef;}#mermaid-svg-yi79e0nj9tmwkUqY .section-3 rect,#mermaid-svg-yi79e0nj9tmwkUqY .section-3 path,#mermaid-svg-yi79e0nj9tmwkUqY .section-3 circle,#mermaid-svg-yi79e0nj9tmwkUqY .section-3 polygon,#mermaid-svg-yi79e0nj9tmwkUqY .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .section-3 text{fill:black;}#mermaid-svg-yi79e0nj9tmwkUqY .node-icon-3{font-size:40px;color:black;}#mermaid-svg-yi79e0nj9tmwkUqY .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .edge-depth-3{stroke-width:5;}#mermaid-svg-yi79e0nj9tmwkUqY .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled,#mermaid-svg-yi79e0nj9tmwkUqY .disabled circle,#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:lightgray;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:#efefef;}#mermaid-svg-yi79e0nj9tmwkUqY .section-4 rect,#mermaid-svg-yi79e0nj9tmwkUqY .section-4 path,#mermaid-svg-yi79e0nj9tmwkUqY .section-4 circle,#mermaid-svg-yi79e0nj9tmwkUqY .section-4 polygon,#mermaid-svg-yi79e0nj9tmwkUqY .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .section-4 text{fill:black;}#mermaid-svg-yi79e0nj9tmwkUqY .node-icon-4{font-size:40px;color:black;}#mermaid-svg-yi79e0nj9tmwkUqY .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .edge-depth-4{stroke-width:2;}#mermaid-svg-yi79e0nj9tmwkUqY .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled,#mermaid-svg-yi79e0nj9tmwkUqY .disabled circle,#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:lightgray;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:#efefef;}#mermaid-svg-yi79e0nj9tmwkUqY .section-5 rect,#mermaid-svg-yi79e0nj9tmwkUqY .section-5 path,#mermaid-svg-yi79e0nj9tmwkUqY .section-5 circle,#mermaid-svg-yi79e0nj9tmwkUqY .section-5 polygon,#mermaid-svg-yi79e0nj9tmwkUqY .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .section-5 text{fill:black;}#mermaid-svg-yi79e0nj9tmwkUqY .node-icon-5{font-size:40px;color:black;}#mermaid-svg-yi79e0nj9tmwkUqY .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .edge-depth-5{stroke-width:-1;}#mermaid-svg-yi79e0nj9tmwkUqY .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled,#mermaid-svg-yi79e0nj9tmwkUqY .disabled circle,#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:lightgray;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:#efefef;}#mermaid-svg-yi79e0nj9tmwkUqY .section-6 rect,#mermaid-svg-yi79e0nj9tmwkUqY .section-6 path,#mermaid-svg-yi79e0nj9tmwkUqY .section-6 circle,#mermaid-svg-yi79e0nj9tmwkUqY .section-6 polygon,#mermaid-svg-yi79e0nj9tmwkUqY .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .section-6 text{fill:black;}#mermaid-svg-yi79e0nj9tmwkUqY .node-icon-6{font-size:40px;color:black;}#mermaid-svg-yi79e0nj9tmwkUqY .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .edge-depth-6{stroke-width:-4;}#mermaid-svg-yi79e0nj9tmwkUqY .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled,#mermaid-svg-yi79e0nj9tmwkUqY .disabled circle,#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:lightgray;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:#efefef;}#mermaid-svg-yi79e0nj9tmwkUqY .section-7 rect,#mermaid-svg-yi79e0nj9tmwkUqY .section-7 path,#mermaid-svg-yi79e0nj9tmwkUqY .section-7 circle,#mermaid-svg-yi79e0nj9tmwkUqY .section-7 polygon,#mermaid-svg-yi79e0nj9tmwkUqY .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .section-7 text{fill:black;}#mermaid-svg-yi79e0nj9tmwkUqY .node-icon-7{font-size:40px;color:black;}#mermaid-svg-yi79e0nj9tmwkUqY .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .edge-depth-7{stroke-width:-7;}#mermaid-svg-yi79e0nj9tmwkUqY .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled,#mermaid-svg-yi79e0nj9tmwkUqY .disabled circle,#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:lightgray;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:#efefef;}#mermaid-svg-yi79e0nj9tmwkUqY .section-8 rect,#mermaid-svg-yi79e0nj9tmwkUqY .section-8 path,#mermaid-svg-yi79e0nj9tmwkUqY .section-8 circle,#mermaid-svg-yi79e0nj9tmwkUqY .section-8 polygon,#mermaid-svg-yi79e0nj9tmwkUqY .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .section-8 text{fill:black;}#mermaid-svg-yi79e0nj9tmwkUqY .node-icon-8{font-size:40px;color:black;}#mermaid-svg-yi79e0nj9tmwkUqY .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .edge-depth-8{stroke-width:-10;}#mermaid-svg-yi79e0nj9tmwkUqY .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled,#mermaid-svg-yi79e0nj9tmwkUqY .disabled circle,#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:lightgray;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:#efefef;}#mermaid-svg-yi79e0nj9tmwkUqY .section-9 rect,#mermaid-svg-yi79e0nj9tmwkUqY .section-9 path,#mermaid-svg-yi79e0nj9tmwkUqY .section-9 circle,#mermaid-svg-yi79e0nj9tmwkUqY .section-9 polygon,#mermaid-svg-yi79e0nj9tmwkUqY .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .section-9 text{fill:black;}#mermaid-svg-yi79e0nj9tmwkUqY .node-icon-9{font-size:40px;color:black;}#mermaid-svg-yi79e0nj9tmwkUqY .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .edge-depth-9{stroke-width:-13;}#mermaid-svg-yi79e0nj9tmwkUqY .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled,#mermaid-svg-yi79e0nj9tmwkUqY .disabled circle,#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:lightgray;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:#efefef;}#mermaid-svg-yi79e0nj9tmwkUqY .section-10 rect,#mermaid-svg-yi79e0nj9tmwkUqY .section-10 path,#mermaid-svg-yi79e0nj9tmwkUqY .section-10 circle,#mermaid-svg-yi79e0nj9tmwkUqY .section-10 polygon,#mermaid-svg-yi79e0nj9tmwkUqY .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .section-10 text{fill:black;}#mermaid-svg-yi79e0nj9tmwkUqY .node-icon-10{font-size:40px;color:black;}#mermaid-svg-yi79e0nj9tmwkUqY .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .edge-depth-10{stroke-width:-16;}#mermaid-svg-yi79e0nj9tmwkUqY .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled,#mermaid-svg-yi79e0nj9tmwkUqY .disabled circle,#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:lightgray;}#mermaid-svg-yi79e0nj9tmwkUqY .disabled text{fill:#efefef;}#mermaid-svg-yi79e0nj9tmwkUqY .section-root rect,#mermaid-svg-yi79e0nj9tmwkUqY .section-root path,#mermaid-svg-yi79e0nj9tmwkUqY .section-root circle,#mermaid-svg-yi79e0nj9tmwkUqY .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-yi79e0nj9tmwkUqY .section-root text{fill:#ffffff;}#mermaid-svg-yi79e0nj9tmwkUqY .section-root span{color:#ffffff;}#mermaid-svg-yi79e0nj9tmwkUqY .section-2 span{color:#ffffff;}#mermaid-svg-yi79e0nj9tmwkUqY .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-yi79e0nj9tmwkUqY .edge{fill:none;}#mermaid-svg-yi79e0nj9tmwkUqY .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-yi79e0nj9tmwkUqY :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
监控与运维
监控体系
集群监控
节点监控
索引监控
性能监控
监控工具
Cat API
Stats API
Kibana监控
Prometheus
Grafana
告警系统
告警规则
告警渠道
告警处理
备份恢复
快照备份
增量备份
恢复策略
灾难恢复
运维任务
滚动重启
索引重建
数据迁移
容量规划
故障处理
集群故障
节点故障
性能问题
数据问题
一、为什么需要监控?
1.1 监控的重要性
及时发现问题:
– 磁盘空间不足
– 内存使用过高
– 查询性能下降
– 节点故障
预防故障:
– 提前扩容
– 优化配置
– 调整资源
1.2 监控维度
┌─────────────────────────────────────────────────────────┐
│ 监控维度 │
├─────────────────────────────────────────────────────────┤
│ │
│ 1. 集群健康 │
│ └─ 状态、节点数、分片状态 │
│ │
│ 2. 性能指标 │
│ └─ QPS、响应时间、吞吐量 │
│ │
│ 3. 资源使用 │
│ └─ CPU、内存、磁盘、网络 │
│ │
│ 4. JVM 指标 │
│ └─ 堆内存、GC 次数、GC 时间 │
│ │
│ 5. 索引指标 │
│ └─ 文档数、大小、查询次数 │
│ │
└─────────────────────────────────────────────────────────┘
二、集群监控
2.1 集群健康检查
# 查看集群健康
GET /_cluster/health
{
"cluster_name": "my-cluster",
"status": "green",
"timed_out": false,
"number_of_nodes": 3,
"number_of_data_nodes": 3,
"active_primary_shards": 10,
"active_shards": 20,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100.0
}
关键指标:
status:green/yellow/red
number_of_nodes:节点数量
active_shards:活跃分片数
unassigned_shards:未分配分片数(应为0)
active_shards_percent:分片分配百分比(应为100%)
2.2 节点信息
# 查看节点列表
GET /_cat/nodes?v
ip heap.percent ram.percent cpu load_1m node.role master name
192.168.1.101 20 50 2 0.5 dm * node-1
192.168.1.102 18 48 1 0.3 dm – node-2
192.168.1.103 22 52 3 0.6 dm – node-3
关键指标:
heap.percent:堆内存使用率(< 75%)
ram.percent:物理内存使用率(< 85%)
cpu:CPU 使用率(< 80%)
load_1m:1分钟负载
2.3 分片状态
# 查看分片分配
GET /_cat/shards?v
index shard prirep state docs store node
products 0 p STARTED 10000 50mb node-1
products 0 r STARTED 10000 50mb node-2
products 1 p STARTED 12000 60mb node-2
products 1 r STARTED 12000 60mb node-3
关键指标:
state:分片状态(应为 STARTED)
docs:文档数量
store:存储大小
2.4 索引统计
# 查看索引统计
GET /products/_stats
{
"indices": {
"products": {
"primaries": {
"docs": {
"count": 100000,
"deleted": 1000
},
"store": {
"size_in_bytes": 524288000
},
"indexing": {
"index_total": 100000,
"index_time_in_millis": 50000
},
"search": {
"query_total": 50000,
"query_time_in_millis": 25000
}
}
}
}
}
三、性能监控
3.1 查询性能
# 查看慢查询
GET /_cat/thread_pool?v
# 查看正在执行的任务
GET /_tasks
# 查看热点线程
GET /_nodes/hot_threads
3.2 JVM 监控
# 查看 JVM 信息
GET /_nodes/stats/jvm
{
"nodes": {
"node-1": {
"jvm": {
"mem": {
"heap_used_in_bytes": 8589934592,
"heap_used_percent": 50,
"heap_max_in_bytes": 17179869184
},
"gc": {
"collectors": {
"young": {
"collection_count": 100,
"collection_time_in_millis": 1000
},
"old": {
"collection_count": 5,
"collection_time_in_millis": 500
}
}
}
}
}
}
}
关键指标:
heap_used_percent:堆内存使用率(< 75%)
gc.collection_count:GC 次数
gc.collection_time_in_millis:GC 时间
3.3 磁盘监控
# 查看磁盘使用
GET /_cat/allocation?v
shards disk.indices disk.used disk.avail disk.total disk.percent node
10 500gb 600gb 400gb 1tb 60 node-1
10 480gb 580gb 420gb 1tb 58 node-2
四、使用 Kibana 监控
4.1 Stack Monitoring
Kibana → Stack Monitoring
功能:
– 集群概览
– 节点监控
– 索引监控
– 性能图表
– 告警配置
4.2 关键指标面板
集群概览:
– 集群状态
– 节点数量
– 索引数量
– 文档数量
性能指标:
– 搜索速率
– 索引速率
– 查询延迟
– 索引延迟
资源使用:
– CPU 使用率
– 内存使用率
– 磁盘使用率
– JVM 堆内存
五、使用 Prometheus + Grafana
5.1 安装 Elasticsearch Exporter
# 下载
wget https://github.com/prometheus-community/elasticsearch_exporter/releases/download/v1.5.0/elasticsearch_exporter-1.5.0.linux-amd64.tar.gz
# 解压
tar -xzf elasticsearch_exporter-1.5.0.linux-amd64.tar.gz
# 启动
./elasticsearch_exporter –es.uri=http://localhost:9200
5.2 配置 Prometheus
# prometheus.yml
scrape_configs:
– job_name: 'elasticsearch'
static_configs:
– targets: ['localhost:9114']
5.3 Grafana 仪表板
导入仪表板:
– Dashboard ID: 2322(Elasticsearch Overview)
– Dashboard ID: 14191(Elasticsearch Cluster)
六、告警配置
6.1 Kibana 告警
// 创建告警规则
POST /_watcher/watch/cluster_health_watch
{
"trigger": {
"schedule": {
"interval": "1m"
}
},
"input": {
"http": {
"request": {
"host": "localhost",
"port": 9200,
"path": "/_cluster/health"
}
}
},
"condition": {
"compare": {
"ctx.payload.status": {
"not_eq": "green"
}
}
},
"actions": {
"send_email": {
"email": {
"to": "admin@example.com",
"subject": "ES 集群状态异常",
"body": "集群状态: {{ctx.payload.status}}"
}
}
}
}
6.2 常见告警规则
1. 集群状态非 green
2. 堆内存使用率 > 85%
3. 磁盘使用率 > 85%
4. 未分配分片数 > 0
5. 查询响应时间 > 1s
6. GC 时间占比 > 10%
七、备份与恢复
7.1 快照仓库配置
# elasticsearch.yml
path.repo: ["/mount/backups"]
创建快照仓库:
PUT /_snapshot/my_backup
{
"type": "fs",
"settings": {
"location": "/mount/backups",
"compress": true
}
}
7.2 创建快照
# 创建快照
PUT /_snapshot/my_backup/snapshot_1
{
"indices": "products,orders",
"ignore_unavailable": true,
"include_global_state": false
}
# 查看快照状态
GET /_snapshot/my_backup/snapshot_1
# 查看所有快照
GET /_snapshot/my_backup/_all
7.3 恢复快照
# 恢复快照
POST /_snapshot/my_backup/snapshot_1/_restore
{
"indices": "products",
"ignore_unavailable": true,
"include_global_state": false,
"rename_pattern": "(.+)",
"rename_replacement": "restored_$1"
}
# 查看恢复状态
GET /_recovery/restored_products
7.4 定时备份
# 使用 cron 定时备份
0 2 * * * curl -X PUT "localhost:9200/_snapshot/my_backup/snapshot_$(date +\\%Y\\%m\\%d)" -H 'Content-Type: application/json' -d'{"indices":"*"}'
八、日志管理
8.1 日志类型
主日志:
– elasticsearch.log
– 记录集群事件
慢查询日志:
– elasticsearch_index_search_slowlog.log
– 记录慢查询
慢索引日志:
– elasticsearch_index_indexing_slowlog.log
– 记录慢索引
GC 日志:
– gc.log
– 记录 GC 信息
8.2 日志配置
# log4j2.properties
logger.action.name = org.elasticsearch.action
logger.action.level = info
appender.rolling.type = RollingFile
appender.rolling.name = rolling
appender.rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}.log
appender.rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}–%d{yyyy–MM–dd}–%i.log.gz
8.3 日志分析
# 查看错误日志
grep "ERROR" elasticsearch.log
# 查看慢查询
grep "took_millis" elasticsearch_index_search_slowlog.log
# 统计错误次数
grep "ERROR" elasticsearch.log | wc -l
九、常见运维任务
9.1 滚动重启
# 1. 禁用分片分配
PUT /_cluster/settings
{
"transient": {
"cluster.routing.allocation.enable": "primaries"
}
}
# 2. 停止节点
systemctl stop elasticsearch
# 3. 升级/维护
# 4. 启动节点
systemctl start elasticsearch
# 5. 等待节点加入集群
GET /_cat/nodes
# 6. 启用分片分配
PUT /_cluster/settings
{
"transient": {
"cluster.routing.allocation.enable": "all"
}
}
# 7. 等待集群恢复 green
GET /_cluster/health?wait_for_status=green&timeout=50s
# 8. 重复步骤 1-7 处理其他节点
9.2 清理旧数据
# 删除旧索引
DELETE /logs-2023-*
# 使用 Curator 自动清理
curator –config config.yml delete_indices.yml
9.3 索引重建
# 1. 创建新索引
PUT /products_v2
# 2. 重建数据
POST /_reindex
{
"source": { "index": "products_v1" },
"dest": { "index": "products_v2" }
}
# 3. 切换别名
POST /_aliases
{
"actions": [
{ "remove": { "index": "products_v1", "alias": "products" }},
{ "add": { "index": "products_v2", "alias": "products" }}
]
}
# 4. 删除旧索引
DELETE /products_v1
十、故障处理详解
10.1 集群状态 Yellow 完整处理流程
原因分析:
1. 副本分片未分配
– 节点数量不足
– 磁盘空间不足
– 分片分配被禁用
2. 节点故障
– 节点离线
– 网络分区
3. 配置问题
– 副本数配置过高
– 分片分配规则限制
详细排查步骤:
# 1. 查看集群健康详情
GET /_cluster/health?level=indices
# 2. 查看未分配的分片
GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED
# 3. 查看分片分配解释
GET /_cluster/allocation/explain
{
"index": "products",
"shard": 0,
"primary": false
}
# 4. 查看节点信息
GET /_cat/nodes?v
# 5. 查看磁盘使用情况
GET /_cat/allocation?v
解决方案:
# 方案 1:减少副本数(临时方案)
PUT /products/_settings
{
"number_of_replicas": 0
}
# 方案 2:增加节点(推荐)
# 启动新节点加入集群
# 方案 3:手动分配分片
POST /_cluster/reroute
{
"commands": [
{
"allocate_replica": {
"index": "products",
"shard": 0,
"node": "node-2"
}
}
]
}
# 方案 4:调整分片分配设置
PUT /_cluster/settings
{
"transient": {
"cluster.routing.allocation.enable": "all",
"cluster.routing.allocation.disk.watermark.low": "90%",
"cluster.routing.allocation.disk.watermark.high": "95%"
}
}
10.2 集群状态 Red 完整处理流程
原因分析:
1. 主分片丢失
– 节点故障导致数据丢失
– 磁盘损坏
2. 索引损坏
– 段文件损坏
– 事务日志损坏
3. 配置错误
– 分片分配被禁用
详细排查步骤:
# 1. 查看 Red 状态的索引
GET /_cat/indices?v&health=red
# 2. 查看未分配的主分片
GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason | grep "p.*UNASSIGNED"
# 3. 查看分片分配解释
GET /_cluster/allocation/explain
{
"index": "products",
"shard": 0,
"primary": true
}
# 4. 查看节点日志
tail -f /var/log/elasticsearch/my-cluster.log | grep ERROR
解决方案:
# 方案 1:从快照恢复(推荐)
POST /_snapshot/my_backup/snapshot_1/_restore
{
"indices": "products",
"ignore_unavailable": true
}
# 方案 2:强制分配空主分片(会丢失数据)
POST /_cluster/reroute
{
"commands": [
{
"allocate_empty_primary": {
"index": "products",
"shard": 0,
"node": "node-1",
"accept_data_loss": true
}
}
]
}
# 方案 3:从副本提升为主分片
POST /_cluster/reroute
{
"commands": [
{
"allocate_stale_primary": {
"index": "products",
"shard": 0,
"node": "node-2",
"accept_data_loss": true
}
}
]
}
# 方案 4:删除损坏的索引(最后手段)
DELETE /products
10.3 磁盘空间不足详细处理
磁盘水位线说明:
Low Watermark(默认 85%):
– 不再分配新分片到该节点
– 现有分片继续工作
High Watermark(默认 90%):
– 尝试将分片迁移到其他节点
– 触发分片重新分配
Flood Stage Watermark(默认 95%):
– 索引变为只读
– 无法写入新数据
– 需要立即处理
详细处理步骤:
# 1. 查看磁盘使用情况
GET /_cat/allocation?v&h=node,disk.used_percent,disk.used,disk.avail,disk.total
# 2. 查看索引大小
GET /_cat/indices?v&h=index,store.size&s=store.size:desc
# 3. 删除不需要的索引
DELETE /logs-2023-01-*
# 4. 合并段(减少段文件数量)
POST /products/_forcemerge?max_num_segments=1
# 5. 清理缓存
POST /_cache/clear
# 6. 清理事务日志
POST /products/_flush
# 7. 如果索引被标记为只读,解除限制
PUT /products/_settings
{
"index.blocks.read_only_allow_delete": null
}
# 8. 调整磁盘水位线(临时方案)
PUT /_cluster/settings
{
"transient": {
"cluster.routing.allocation.disk.watermark.low": "90%",
"cluster.routing.allocation.disk.watermark.high": "95%",
"cluster.routing.allocation.disk.watermark.flood_stage": "97%"
}
}
# 9. 增加磁盘空间(长期方案)
# – 添加新磁盘
# – 添加新节点
# – 使用 ILM 自动清理旧数据
10.4 内存溢出(OOM)详细处理
OOM 原因分析:
1. 堆内存不足
– 配置过小
– 查询消耗过多内存
– 聚合操作过大
2. 字段数据缓存过大
– 大量排序操作
– 大量聚合操作
3. 内存泄漏
– 长时间运行的查询
– 未释放的资源
详细处理步骤:
# 1. 查看堆内存使用情况
GET /_nodes/stats/jvm?human
# 2. 查看字段数据缓存
GET /_nodes/stats/indices/fielddata?human
# 3. 查看断路器状态
GET /_nodes/stats/breaker?human
# 4. 清理字段数据缓存
POST /_cache/clear?fielddata=true
# 5. 清理查询缓存
POST /_cache/clear?query=true
# 6. 清理请求缓存
POST /_cache/clear?request=true
# 7. 取消长时间运行的任务
GET /_tasks?detailed=true&actions=*search*
POST /_tasks/{task_id}/_cancel
# 8. 调整堆内存(需要重启)
# jvm.options
-Xms16g
-Xmx16g
# 9. 调整断路器限制
PUT /_cluster/settings
{
"transient": {
"indices.breaker.fielddata.limit": "40%",
"indices.breaker.request.limit": "60%",
"indices.breaker.total.limit": "95%"
}
}
# 10. 优化查询
# – 使用 filter 代替 query
# – 限制聚合的 bucket 数量
# – 使用 doc_values 代替 fielddata
# – 避免深度分页
10.5 查询性能问题
性能问题排查:
# 1. 查看慢查询日志
tail -f logs/my-cluster_index_search_slowlog.log
# 2. 查看线程池状态
GET /_cat/thread_pool?v&h=node_name,name,active,queue,rejected,completed
# 3. 查看热点线程
GET /_nodes/hot_threads
# 4. 使用 Profile API 分析查询
GET /products/_search
{
"profile": true,
"query": {
"match": { "name": "手机" }
}
}
# 5. 查看查询缓存命中率
GET /_nodes/stats/indices/query_cache?human
性能优化方案:
# 1. 启用查询缓存
PUT /products/_settings
{
"index.queries.cache.enabled": true
}
# 2. 使用 filter context
{
"query": {
"bool": {
"must": [
{ "match": { "name": "手机" }}
],
"filter": [
{ "term": { "onSale": true }}
]
}
}
}
# 3. 减少返回字段
{
"query": { "match_all": {} },
"_source": ["name", "price"]
}
# 4. 使用 search_after 代替深度分页
{
"size": 10,
"search_after": [3999, "last_id"],
"sort": [
{ "price": "asc" },
{ "_id": "asc" }
]
}
# 5. 增加刷新间隔
PUT /products/_settings
{
"refresh_interval": "30s"
}
# 6. 增加搜索线程池
# elasticsearch.yml
thread_pool:
search:
size: 30
queue_size: 1000
10.6 索引速度慢
索引性能排查:
# 1. 查看慢索引日志
tail -f logs/my-cluster_index_indexing_slowlog.log
# 2. 查看索引统计
GET /products/_stats/indexing
# 3. 查看段信息
GET /_cat/segments/products?v
# 4. 查看合并统计
GET /products/_stats/merge
索引优化方案:
# 1. 批量索引时禁用刷新
PUT /products/_settings
{
"refresh_interval": "-1",
"number_of_replicas": 0
}
# 批量导入数据…
# 恢复设置
POST /products/_refresh
PUT /products/_settings
{
"refresh_interval": "1s",
"number_of_replicas": 1
}
# 2. 使用 Bulk API
POST /_bulk
{"index": {"_index": "products", "_id": "1"}}
{"name": "产品1"}
{"index": {"_index": "products", "_id": "2"}}
{"name": "产品2"}
# 3. 增加索引缓冲区
PUT /_cluster/settings
{
"transient": {
"indices.memory.index_buffer_size": "20%"
}
}
# 4. 调整合并策略
PUT /products/_settings
{
"index.merge.scheduler.max_thread_count": 1
}
# 5. 增加写入线程池
# elasticsearch.yml
thread_pool:
write:
size: 8
queue_size: 1000
十一、自动化运维
11.1 使用 Curator 自动管理索引
安装 Curator:
pip install elasticsearch-curator
配置文件(config.yml):
client:
hosts:
– localhost
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: False
http_auth:
timeout: 30
master_only: False
logging:
loglevel: INFO
logfile:
logformat: default
blacklist: ['elasticsearch', 'urllib3']
删除旧索引(delete_indices.yml):
actions:
1:
action: delete_indices
description: >–
删除超过 30 天的日志索引
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: False
filters:
– filtertype: pattern
kind: prefix
value: logs–
exclude:
– filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 30
exclude:
关闭旧索引(close_indices.yml):
actions:
1:
action: close
description: >–
关闭超过 7 天的日志索引
options:
ignore_empty_list: True
delete_aliases: False
timeout_override:
continue_if_exception: False
disable_action: False
filters:
– filtertype: pattern
kind: prefix
value: logs–
exclude:
– filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 7
exclude:
定时执行(crontab):
# 每天凌晨 2 点执行
0 2 * * * curator –config /etc/curator/config.yml /etc/curator/delete_indices.yml
11.2 使用 ILM 自动管理索引生命周期
创建 ILM 策略:
PUT /_ilm/policy/logs_policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "50GB",
"max_age": "1d",
"max_docs": 10000000
},
"set_priority": {
"priority": 100
}
}
},
"warm": {
"min_age": "7d",
"actions": {
"shrink": {
"number_of_shards": 1
},
"forcemerge": {
"max_num_segments": 1
},
"set_priority": {
"priority": 50
}
}
},
"cold": {
"min_age": "30d",
"actions": {
"freeze": {},
"set_priority": {
"priority": 0
}
}
},
"delete": {
"min_age": "90d",
"actions": {
"delete": {}
}
}
}
}
}
应用 ILM 策略:
PUT /_index_template/logs_template
{
"index_patterns": ["logs-*"],
"template": {
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"index.lifecycle.name": "logs_policy",
"index.lifecycle.rollover_alias": "logs"
}
}
}
PUT /logs–000001
{
"aliases": {
"logs": {
"is_write_index": true
}
}
}
11.3 监控脚本
集群健康检查脚本:
#!/bin/bash
# cluster_health_check.sh
ES_HOST="localhost:9200"
ALERT_EMAIL="admin@example.com"
# 检查集群状态
STATUS=$(curl -s "http://${ES_HOST}/_cluster/health" | jq -r '.status')
if [ "$STATUS" != "green" ]; then
echo "集群状态异常: $STATUS" | mail -s "ES 集群告警" $ALERT_EMAIL
fi
# 检查磁盘使用率
DISK_USAGE=$(curl -s "http://${ES_HOST}/_cat/allocation?h=disk.percent" | sort -rn | head -1)
if [ "$DISK_USAGE" -gt 85 ]; then
echo "磁盘使用率过高: ${DISK_USAGE}%" | mail -s "ES 磁盘告警" $ALERT_EMAIL
fi
# 检查堆内存使用率
HEAP_USAGE=$(curl -s "http://${ES_HOST}/_cat/nodes?h=heap.percent" | sort -rn | head -1)
if [ "$HEAP_USAGE" -gt 85 ]; then
echo "堆内存使用率过高: ${HEAP_USAGE}%" | mail -s "ES 内存告警" $ALERT_EMAIL
fi
定时执行:
# crontab -e
*/5 * * * * /path/to/cluster_health_check.sh
十二、容量规划
12.1 数据增长预测
当前数据量:1TB
日增长量:10GB
保留时间:90天
预测容量 = 1TB + 10GB × 90 = 1.9TB
考虑副本(1个)= 1.9TB × 2 = 3.8TB
考虑索引开销(1.5倍)= 3.8TB × 1.5 = 5.7TB
预留空间(20%)= 5.7TB × 1.2 = 6.84TB
建议总容量:7TB
12.2 节点数量规划
单节点存储容量:1TB
总容量需求:7TB
副本数:1
数据节点数 = 7TB / 1TB × (1 + 1) = 14 个节点
考虑高可用:
– Master 节点:3 个
– Data 节点:14 个
– Coordinating 节点:2 个
总节点数:19 个
12.3 分片数量规划
数据总量:7TB
单分片目标大小:30GB
主分片数 = 7TB / 30GB = 233 个分片
分配到索引:
– 如果有 10 个索引
– 每个索引:23 个主分片
学习检查清单
- 深入掌握集群监控的所有方法
- 理解所有关键性能指标和告警阈值
- 能够配置完整的告警系统
- 掌握备份恢复的最佳实践
- 能够处理所有常见故障
- 掌握自动化运维工具的使用
- 能够进行容量规划
- 了解日志管理和分析
- 完成所有实战练习
下一步
下一节我们将总结 Elasticsearch 的常见问题与解决方案,帮助你快速定位和解决问题。


