ZooKeeper节点宕机处理机制深度解析：故障检测与恢复全流程

ZooKeeper节点宕机处理机制深度解析：故障检测与恢复全流程

• • 一、节点宕机的类型与影响
  • • 1.1 宕机类型分类
    • 1.2 宕机检测机制
  • 二、Follower节点宕机处理
  • • 2.1 处理流程图
    • 2.2 Follower宕机的影响分析
    • 2.3 Follower宕机的处理细节
  • 三、Leader节点宕机处理
  • • 3.1 处理流程图
    • 3.2 Leader选举详细过程
    • • 步骤1：进入LOOKING状态
      • 步骤2：发起投票
      • 步骤3：比较投票规则
      • 步骤4：统计投票结果
      • 步骤5：数据同步
    • 3.3 新旧Leader的epoch变化
  • 四、过半机制与可用性
  • • 4.1 过半机制的核心作用
    • 4.2 不同规模集群的容错能力
    • 4.3 3节点集群的故障场景分析
  • 五、Observer节点的特殊处理
  • • 5.1 Observer宕机的影响
    • 5.2 Observer宕机的处理
  • 六、故障恢复与数据一致性
  • • 6.1 宕机节点重新加入
    • 6.2 一致性保证
  • 七、最佳实践与监控
  • • 7.1 集群规划建议
    • 7.2 监控指标
  • 八、总结
  • • 8.1 宕机处理的核心机制
    • 8.2 宕机处理流程回顾
    • 8.3 一句话总结

摘要：在分布式系统中，节点宕机是常态而非异常。ZooKeeper作为高可用的协调服务，设计了完善的故障处理机制。当节点宕机时，系统会自动进行故障检测、Leader重新选举和数据同步，确保集群持续可用。本文将深入剖析ZooKeeper处理节点宕机的完整流程。

一、节点宕机的类型与影响

1.1 宕机类型分类

根据宕机节点的角色不同，影响范围也不同：

#mermaid-svg-11mksMfBfXS6ilxg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-11mksMfBfXS6ilxg .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-11mksMfBfXS6ilxg .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-11mksMfBfXS6ilxg .error-icon{fill:#552222;}#mermaid-svg-11mksMfBfXS6ilxg .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-11mksMfBfXS6ilxg .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-11mksMfBfXS6ilxg .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-11mksMfBfXS6ilxg .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-11mksMfBfXS6ilxg .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-11mksMfBfXS6ilxg .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-11mksMfBfXS6ilxg .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-11mksMfBfXS6ilxg .marker{fill:#333333;stroke:#333333;}#mermaid-svg-11mksMfBfXS6ilxg .marker.cross{stroke:#333333;}#mermaid-svg-11mksMfBfXS6ilxg svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-11mksMfBfXS6ilxg p{margin:0;}#mermaid-svg-11mksMfBfXS6ilxg .edge{stroke-width:3;}#mermaid-svg-11mksMfBfXS6ilxg .section–1 rect,#mermaid-svg-11mksMfBfXS6ilxg .section–1 path,#mermaid-svg-11mksMfBfXS6ilxg .section–1 circle,#mermaid-svg-11mksMfBfXS6ilxg .section–1 polygon,#mermaid-svg-11mksMfBfXS6ilxg .section–1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .section–1 text{fill:#ffffff;}#mermaid-svg-11mksMfBfXS6ilxg .node-icon–1{font-size:40px;color:#ffffff;}#mermaid-svg-11mksMfBfXS6ilxg .section-edge–1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .edge-depth–1{stroke-width:17;}#mermaid-svg-11mksMfBfXS6ilxg .section–1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-11mksMfBfXS6ilxg .disabled,#mermaid-svg-11mksMfBfXS6ilxg .disabled circle,#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:lightgray;}#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:#efefef;}#mermaid-svg-11mksMfBfXS6ilxg .section-0 rect,#mermaid-svg-11mksMfBfXS6ilxg .section-0 path,#mermaid-svg-11mksMfBfXS6ilxg .section-0 circle,#mermaid-svg-11mksMfBfXS6ilxg .section-0 polygon,#mermaid-svg-11mksMfBfXS6ilxg .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-11mksMfBfXS6ilxg .section-0 text{fill:black;}#mermaid-svg-11mksMfBfXS6ilxg .node-icon-0{font-size:40px;color:black;}#mermaid-svg-11mksMfBfXS6ilxg .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-11mksMfBfXS6ilxg .edge-depth-0{stroke-width:14;}#mermaid-svg-11mksMfBfXS6ilxg .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-11mksMfBfXS6ilxg .disabled,#mermaid-svg-11mksMfBfXS6ilxg .disabled circle,#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:lightgray;}#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:#efefef;}#mermaid-svg-11mksMfBfXS6ilxg .section-1 rect,#mermaid-svg-11mksMfBfXS6ilxg .section-1 path,#mermaid-svg-11mksMfBfXS6ilxg .section-1 circle,#mermaid-svg-11mksMfBfXS6ilxg .section-1 polygon,#mermaid-svg-11mksMfBfXS6ilxg .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .section-1 text{fill:black;}#mermaid-svg-11mksMfBfXS6ilxg .node-icon-1{font-size:40px;color:black;}#mermaid-svg-11mksMfBfXS6ilxg .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .edge-depth-1{stroke-width:11;}#mermaid-svg-11mksMfBfXS6ilxg .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-11mksMfBfXS6ilxg .disabled,#mermaid-svg-11mksMfBfXS6ilxg .disabled circle,#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:lightgray;}#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:#efefef;}#mermaid-svg-11mksMfBfXS6ilxg .section-2 rect,#mermaid-svg-11mksMfBfXS6ilxg .section-2 path,#mermaid-svg-11mksMfBfXS6ilxg .section-2 circle,#mermaid-svg-11mksMfBfXS6ilxg .section-2 polygon,#mermaid-svg-11mksMfBfXS6ilxg .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .section-2 text{fill:#ffffff;}#mermaid-svg-11mksMfBfXS6ilxg .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-11mksMfBfXS6ilxg .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .edge-depth-2{stroke-width:8;}#mermaid-svg-11mksMfBfXS6ilxg .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-11mksMfBfXS6ilxg .disabled,#mermaid-svg-11mksMfBfXS6ilxg .disabled circle,#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:lightgray;}#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:#efefef;}#mermaid-svg-11mksMfBfXS6ilxg .section-3 rect,#mermaid-svg-11mksMfBfXS6ilxg .section-3 path,#mermaid-svg-11mksMfBfXS6ilxg .section-3 circle,#mermaid-svg-11mksMfBfXS6ilxg .section-3 polygon,#mermaid-svg-11mksMfBfXS6ilxg .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .section-3 text{fill:black;}#mermaid-svg-11mksMfBfXS6ilxg .node-icon-3{font-size:40px;color:black;}#mermaid-svg-11mksMfBfXS6ilxg .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .edge-depth-3{stroke-width:5;}#mermaid-svg-11mksMfBfXS6ilxg .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-11mksMfBfXS6ilxg .disabled,#mermaid-svg-11mksMfBfXS6ilxg .disabled circle,#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:lightgray;}#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:#efefef;}#mermaid-svg-11mksMfBfXS6ilxg .section-4 rect,#mermaid-svg-11mksMfBfXS6ilxg .section-4 path,#mermaid-svg-11mksMfBfXS6ilxg .section-4 circle,#mermaid-svg-11mksMfBfXS6ilxg .section-4 polygon,#mermaid-svg-11mksMfBfXS6ilxg .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .section-4 text{fill:black;}#mermaid-svg-11mksMfBfXS6ilxg .node-icon-4{font-size:40px;color:black;}#mermaid-svg-11mksMfBfXS6ilxg .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .edge-depth-4{stroke-width:2;}#mermaid-svg-11mksMfBfXS6ilxg .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-11mksMfBfXS6ilxg .disabled,#mermaid-svg-11mksMfBfXS6ilxg .disabled circle,#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:lightgray;}#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:#efefef;}#mermaid-svg-11mksMfBfXS6ilxg .section-5 rect,#mermaid-svg-11mksMfBfXS6ilxg .section-5 path,#mermaid-svg-11mksMfBfXS6ilxg .section-5 circle,#mermaid-svg-11mksMfBfXS6ilxg .section-5 polygon,#mermaid-svg-11mksMfBfXS6ilxg .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .section-5 text{fill:black;}#mermaid-svg-11mksMfBfXS6ilxg .node-icon-5{font-size:40px;color:black;}#mermaid-svg-11mksMfBfXS6ilxg .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .edge-depth-5{stroke-width:-1;}#mermaid-svg-11mksMfBfXS6ilxg .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-11mksMfBfXS6ilxg .disabled,#mermaid-svg-11mksMfBfXS6ilxg .disabled circle,#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:lightgray;}#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:#efefef;}#mermaid-svg-11mksMfBfXS6ilxg .section-6 rect,#mermaid-svg-11mksMfBfXS6ilxg .section-6 path,#mermaid-svg-11mksMfBfXS6ilxg .section-6 circle,#mermaid-svg-11mksMfBfXS6ilxg .section-6 polygon,#mermaid-svg-11mksMfBfXS6ilxg .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .section-6 text{fill:black;}#mermaid-svg-11mksMfBfXS6ilxg .node-icon-6{font-size:40px;color:black;}#mermaid-svg-11mksMfBfXS6ilxg .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .edge-depth-6{stroke-width:-4;}#mermaid-svg-11mksMfBfXS6ilxg .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-11mksMfBfXS6ilxg .disabled,#mermaid-svg-11mksMfBfXS6ilxg .disabled circle,#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:lightgray;}#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:#efefef;}#mermaid-svg-11mksMfBfXS6ilxg .section-7 rect,#mermaid-svg-11mksMfBfXS6ilxg .section-7 path,#mermaid-svg-11mksMfBfXS6ilxg .section-7 circle,#mermaid-svg-11mksMfBfXS6ilxg .section-7 polygon,#mermaid-svg-11mksMfBfXS6ilxg .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .section-7 text{fill:black;}#mermaid-svg-11mksMfBfXS6ilxg .node-icon-7{font-size:40px;color:black;}#mermaid-svg-11mksMfBfXS6ilxg .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .edge-depth-7{stroke-width:-7;}#mermaid-svg-11mksMfBfXS6ilxg .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-11mksMfBfXS6ilxg .disabled,#mermaid-svg-11mksMfBfXS6ilxg .disabled circle,#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:lightgray;}#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:#efefef;}#mermaid-svg-11mksMfBfXS6ilxg .section-8 rect,#mermaid-svg-11mksMfBfXS6ilxg .section-8 path,#mermaid-svg-11mksMfBfXS6ilxg .section-8 circle,#mermaid-svg-11mksMfBfXS6ilxg .section-8 polygon,#mermaid-svg-11mksMfBfXS6ilxg .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .section-8 text{fill:black;}#mermaid-svg-11mksMfBfXS6ilxg .node-icon-8{font-size:40px;color:black;}#mermaid-svg-11mksMfBfXS6ilxg .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .edge-depth-8{stroke-width:-10;}#mermaid-svg-11mksMfBfXS6ilxg .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-11mksMfBfXS6ilxg .disabled,#mermaid-svg-11mksMfBfXS6ilxg .disabled circle,#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:lightgray;}#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:#efefef;}#mermaid-svg-11mksMfBfXS6ilxg .section-9 rect,#mermaid-svg-11mksMfBfXS6ilxg .section-9 path,#mermaid-svg-11mksMfBfXS6ilxg .section-9 circle,#mermaid-svg-11mksMfBfXS6ilxg .section-9 polygon,#mermaid-svg-11mksMfBfXS6ilxg .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .section-9 text{fill:black;}#mermaid-svg-11mksMfBfXS6ilxg .node-icon-9{font-size:40px;color:black;}#mermaid-svg-11mksMfBfXS6ilxg .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .edge-depth-9{stroke-width:-13;}#mermaid-svg-11mksMfBfXS6ilxg .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-11mksMfBfXS6ilxg .disabled,#mermaid-svg-11mksMfBfXS6ilxg .disabled circle,#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:lightgray;}#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:#efefef;}#mermaid-svg-11mksMfBfXS6ilxg .section-10 rect,#mermaid-svg-11mksMfBfXS6ilxg .section-10 path,#mermaid-svg-11mksMfBfXS6ilxg .section-10 circle,#mermaid-svg-11mksMfBfXS6ilxg .section-10 polygon,#mermaid-svg-11mksMfBfXS6ilxg .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .section-10 text{fill:black;}#mermaid-svg-11mksMfBfXS6ilxg .node-icon-10{font-size:40px;color:black;}#mermaid-svg-11mksMfBfXS6ilxg .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .edge-depth-10{stroke-width:-16;}#mermaid-svg-11mksMfBfXS6ilxg .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-11mksMfBfXS6ilxg .disabled,#mermaid-svg-11mksMfBfXS6ilxg .disabled circle,#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:lightgray;}#mermaid-svg-11mksMfBfXS6ilxg .disabled text{fill:#efefef;}#mermaid-svg-11mksMfBfXS6ilxg .section-root rect,#mermaid-svg-11mksMfBfXS6ilxg .section-root path,#mermaid-svg-11mksMfBfXS6ilxg .section-root circle,#mermaid-svg-11mksMfBfXS6ilxg .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-11mksMfBfXS6ilxg .section-root text{fill:#ffffff;}#mermaid-svg-11mksMfBfXS6ilxg .section-root span{color:#ffffff;}#mermaid-svg-11mksMfBfXS6ilxg .section-2 span{color:#ffffff;}#mermaid-svg-11mksMfBfXS6ilxg .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-11mksMfBfXS6ilxg .edge{fill:none;}#mermaid-svg-11mksMfBfXS6ilxg .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-11mksMfBfXS6ilxg :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

节点宕机处理

Follower宕机

影响：读能力下降

处理：自动剔除，业务无感知

Leader宕机

影响：写服务中断

处理：触发选举，恢复写服务

Observer宕机

影响：无影响

处理：无需特殊处理

1.2 宕机检测机制

ZooKeeper通过两种方式检测节点宕机：

# zoo.cfg中的超时配置
tickTime=2000 # 基本时间单位（毫秒）
syncLimit=5 # 心跳超时限制（5 * tickTime = 10秒）
# 如果10秒未收到Follower的心跳，Leader判定该节点宕机

二、Follower节点宕机处理

2.1 处理流程图

Follower3

Follower2

Follower1(宕机)

Leader

Follower3

Follower2

Follower1(宕机)

Leader

#mermaid-svg-3teNul1eoeqARoBC{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-3teNul1eoeqARoBC .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-3teNul1eoeqARoBC .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-3teNul1eoeqARoBC .error-icon{fill:#552222;}#mermaid-svg-3teNul1eoeqARoBC .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-3teNul1eoeqARoBC .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-3teNul1eoeqARoBC .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-3teNul1eoeqARoBC .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-3teNul1eoeqARoBC .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-3teNul1eoeqARoBC .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-3teNul1eoeqARoBC .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-3teNul1eoeqARoBC .marker{fill:#333333;stroke:#333333;}#mermaid-svg-3teNul1eoeqARoBC .marker.cross{stroke:#333333;}#mermaid-svg-3teNul1eoeqARoBC svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-3teNul1eoeqARoBC p{margin:0;}#mermaid-svg-3teNul1eoeqARoBC .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-3teNul1eoeqARoBC text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-3teNul1eoeqARoBC .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-3teNul1eoeqARoBC .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-3teNul1eoeqARoBC .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-3teNul1eoeqARoBC .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-3teNul1eoeqARoBC #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-3teNul1eoeqARoBC .sequenceNumber{fill:white;}#mermaid-svg-3teNul1eoeqARoBC #sequencenumber{fill:#333;}#mermaid-svg-3teNul1eoeqARoBC #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-3teNul1eoeqARoBC .messageText{fill:#333;stroke:none;}#mermaid-svg-3teNul1eoeqARoBC .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-3teNul1eoeqARoBC .labelText,#mermaid-svg-3teNul1eoeqARoBC .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-3teNul1eoeqARoBC .loopText,#mermaid-svg-3teNul1eoeqARoBC .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-3teNul1eoeqARoBC .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-3teNul1eoeqARoBC .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-3teNul1eoeqARoBC .noteText,#mermaid-svg-3teNul1eoeqARoBC .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-3teNul1eoeqARoBC .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-3teNul1eoeqARoBC .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-3teNul1eoeqARoBC .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-3teNul1eoeqARoBC .actorPopupMenu{position:absolute;}#mermaid-svg-3teNul1eoeqARoBC .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-3teNul1eoeqARoBC .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-3teNul1eoeqARoBC .actor-man circle,#mermaid-svg-3teNul1eoeqARoBC line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-3teNul1eoeqARoBC :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

节点宕机，无响应

集群成员从3变为2

过半要求不变(仍需2/3)

只要还有2个节点，集群可用

心跳检测

多次重试超时

判定Follower1宕机

更新集群成员信息

更新集群成员信息

正常请求

正常请求

2.2 Follower宕机的影响分析

2.3 Follower宕机的处理细节

当Follower宕机时，Leader会执行以下操作：

// ZooKeeper源码中的Follower故障处理
public class Leader {

// 处理Follower心跳超时
private void handleFollowerTimeout(LearnerHandler handler) {
// 1. 关闭与该Follower的连接
handler.shutdown();

// 2. 从学习者列表中移除
learners.remove(handler);

// 3. 更新forwardingFollowers（用于事务转发）
forwardingFollowers.remove(handler);

// 4. 记录日志
LOG.info("Follower " + handler.getSid() + " has been removed");

// 5. 继续处理请求（只要还有过半节点）
// 无需触发选举，因为Leader仍然存在
}
}

三、Leader节点宕机处理

3.1 处理流程图

#mermaid-svg-hGuZgb3sZmBKqUXJ{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-hGuZgb3sZmBKqUXJ .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-hGuZgb3sZmBKqUXJ .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-hGuZgb3sZmBKqUXJ .error-icon{fill:#552222;}#mermaid-svg-hGuZgb3sZmBKqUXJ .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-hGuZgb3sZmBKqUXJ .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-hGuZgb3sZmBKqUXJ .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-hGuZgb3sZmBKqUXJ .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-hGuZgb3sZmBKqUXJ .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-hGuZgb3sZmBKqUXJ .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-hGuZgb3sZmBKqUXJ .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-hGuZgb3sZmBKqUXJ .marker{fill:#333333;stroke:#333333;}#mermaid-svg-hGuZgb3sZmBKqUXJ .marker.cross{stroke:#333333;}#mermaid-svg-hGuZgb3sZmBKqUXJ svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-hGuZgb3sZmBKqUXJ p{margin:0;}#mermaid-svg-hGuZgb3sZmBKqUXJ .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-hGuZgb3sZmBKqUXJ .cluster-label text{fill:#333;}#mermaid-svg-hGuZgb3sZmBKqUXJ .cluster-label span{color:#333;}#mermaid-svg-hGuZgb3sZmBKqUXJ .cluster-label span p{background-color:transparent;}#mermaid-svg-hGuZgb3sZmBKqUXJ .label text,#mermaid-svg-hGuZgb3sZmBKqUXJ span{fill:#333;color:#333;}#mermaid-svg-hGuZgb3sZmBKqUXJ .node rect,#mermaid-svg-hGuZgb3sZmBKqUXJ .node circle,#mermaid-svg-hGuZgb3sZmBKqUXJ .node ellipse,#mermaid-svg-hGuZgb3sZmBKqUXJ .node polygon,#mermaid-svg-hGuZgb3sZmBKqUXJ .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-hGuZgb3sZmBKqUXJ .rough-node .label text,#mermaid-svg-hGuZgb3sZmBKqUXJ .node .label text,#mermaid-svg-hGuZgb3sZmBKqUXJ .image-shape .label,#mermaid-svg-hGuZgb3sZmBKqUXJ .icon-shape .label{text-anchor:middle;}#mermaid-svg-hGuZgb3sZmBKqUXJ .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-hGuZgb3sZmBKqUXJ .rough-node .label,#mermaid-svg-hGuZgb3sZmBKqUXJ .node .label,#mermaid-svg-hGuZgb3sZmBKqUXJ .image-shape .label,#mermaid-svg-hGuZgb3sZmBKqUXJ .icon-shape .label{text-align:center;}#mermaid-svg-hGuZgb3sZmBKqUXJ .node.clickable{cursor:pointer;}#mermaid-svg-hGuZgb3sZmBKqUXJ .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-hGuZgb3sZmBKqUXJ .arrowheadPath{fill:#333333;}#mermaid-svg-hGuZgb3sZmBKqUXJ .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-hGuZgb3sZmBKqUXJ .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-hGuZgb3sZmBKqUXJ .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hGuZgb3sZmBKqUXJ .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-hGuZgb3sZmBKqUXJ .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hGuZgb3sZmBKqUXJ .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-hGuZgb3sZmBKqUXJ .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-hGuZgb3sZmBKqUXJ .cluster text{fill:#333;}#mermaid-svg-hGuZgb3sZmBKqUXJ .cluster span{color:#333;}#mermaid-svg-hGuZgb3sZmBKqUXJ div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-hGuZgb3sZmBKqUXJ .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-hGuZgb3sZmBKqUXJ rect.text{fill:none;stroke-width:0;}#mermaid-svg-hGuZgb3sZmBKqUXJ .icon-shape,#mermaid-svg-hGuZgb3sZmBKqUXJ .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hGuZgb3sZmBKqUXJ .icon-shape p,#mermaid-svg-hGuZgb3sZmBKqUXJ .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-hGuZgb3sZmBKqUXJ .icon-shape rect,#mermaid-svg-hGuZgb3sZmBKqUXJ .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hGuZgb3sZmBKqUXJ .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-hGuZgb3sZmBKqUXJ .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-hGuZgb3sZmBKqUXJ :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Leader宕机

剩余Follower检测到连接断开

进入LOOKING状态

开始Leader选举

发起投票(myid, lastZxid)

比较投票

选出新Leader

新Leader更新epoch

新Leader与Follower数据同步

恢复服务

3.2 Leader选举详细过程

步骤1：进入LOOKING状态

所有剩余节点检测到Leader失联后，将自己的状态改为LOOKING，准备选举。

步骤2：发起投票

每个节点广播投票信息(myid, lastZxid)：

步骤3：比较投票规则

public Vote compareVotes(Vote myVote, Vote otherVote) {
// 规则1：优先比较ZXID（数据新旧）
if (otherVote.zxid > myVote.zxid) {
return otherVote; // 数据更新的节点优先
} else if (otherVote.zxid < myVote.zxid) {
return myVote; // 我的数据更新
} else {
// 规则2：ZXID相同时比较myid（服务器ID）
if (otherVote.myid > myVote.myid) {
return otherVote; // myid大的优先
} else {
return myVote;
}
}
}

步骤4：统计投票结果

渲染错误: Mermaid 渲染失败: Parse error on line 3: …" A[Server2 (ZXID=0x100000012, m ———————-^ Expecting 'SQE', 'DOUBLECIRCLEEND', 'PE', '-)', 'STADIUMEND', 'SUBROUTINEEND', 'PIPE', 'CYLINDEREND', 'DIAMOND_STOP', 'TAGEND', 'TRAPEND', 'INVTRAPEND', 'UNICODE_TEXT', 'TEXT', 'TAGSTART', got 'PS'

步骤5：数据同步

新Leader选举完成后，需要与所有Follower进行数据同步：

Follower(Server4)

Follower(Server2)

新Leader(Server3)

Follower(Server4)

Follower(Server2)

新Leader(Server3)

#mermaid-svg-H8AUfKzaIy6ddnOr{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-H8AUfKzaIy6ddnOr .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-H8AUfKzaIy6ddnOr .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-H8AUfKzaIy6ddnOr .error-icon{fill:#552222;}#mermaid-svg-H8AUfKzaIy6ddnOr .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-H8AUfKzaIy6ddnOr .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-H8AUfKzaIy6ddnOr .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-H8AUfKzaIy6ddnOr .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-H8AUfKzaIy6ddnOr .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-H8AUfKzaIy6ddnOr .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-H8AUfKzaIy6ddnOr .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-H8AUfKzaIy6ddnOr .marker{fill:#333333;stroke:#333333;}#mermaid-svg-H8AUfKzaIy6ddnOr .marker.cross{stroke:#333333;}#mermaid-svg-H8AUfKzaIy6ddnOr svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-H8AUfKzaIy6ddnOr p{margin:0;}#mermaid-svg-H8AUfKzaIy6ddnOr .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-H8AUfKzaIy6ddnOr text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-H8AUfKzaIy6ddnOr .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-H8AUfKzaIy6ddnOr .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-H8AUfKzaIy6ddnOr .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-H8AUfKzaIy6ddnOr .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-H8AUfKzaIy6ddnOr #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-H8AUfKzaIy6ddnOr .sequenceNumber{fill:white;}#mermaid-svg-H8AUfKzaIy6ddnOr #sequencenumber{fill:#333;}#mermaid-svg-H8AUfKzaIy6ddnOr #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-H8AUfKzaIy6ddnOr .messageText{fill:#333;stroke:none;}#mermaid-svg-H8AUfKzaIy6ddnOr .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-H8AUfKzaIy6ddnOr .labelText,#mermaid-svg-H8AUfKzaIy6ddnOr .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-H8AUfKzaIy6ddnOr .loopText,#mermaid-svg-H8AUfKzaIy6ddnOr .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-H8AUfKzaIy6ddnOr .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-H8AUfKzaIy6ddnOr .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-H8AUfKzaIy6ddnOr .noteText,#mermaid-svg-H8AUfKzaIy6ddnOr .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-H8AUfKzaIy6ddnOr .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-H8AUfKzaIy6ddnOr .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-H8AUfKzaIy6ddnOr .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-H8AUfKzaIy6ddnOr .actorPopupMenu{position:absolute;}#mermaid-svg-H8AUfKzaIy6ddnOr .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-H8AUfKzaIy6ddnOr .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-H8AUfKzaIy6ddnOr .actor-man circle,#mermaid-svg-H8AUfKzaIy6ddnOr line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-H8AUfKzaIy6ddnOr :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

收到过半ACK

FOLLOWERINFO(lastZxid=0x100000012)

FOLLOWERINFO(lastZxid=0x100000010)

计算同步策略

DIFF同步（数据一致）

SNAP同步（落后较多）

ACK

ACK

UPTODATE

UPTODATE

状态变LEADING

状态变FOLLOWING

状态变FOLLOWING

3.3 新旧Leader的epoch变化

// Leader切换时的epoch变化
public class EpochChange {
// 旧Leader时：epoch=1
// 事务ZXID范围：0x100000001 ~ 0x100000012

// 新Leader选举完成后，epoch自增到2
// 第一个事务ZXID: 0x200000001
// epoch保证了新旧Leader的事务不会混淆
}

四、过半机制与可用性

4.1 过半机制的核心作用

public class MajorityMechanism {

/**
* 判断集群是否可用
* @param aliveNodes 存活节点数
* @param totalNodes 总节点数
*/
public boolean isClusterAvailable(int aliveNodes, int totalNodes) {
int majority = totalNodes / 2 + 1;
return aliveNodes >= majority;
}

// 示例：
// 3节点集群：需要至少2个节点存活
// 5节点集群：需要至少3个节点存活
}

4.2 不同规模集群的容错能力

4.3 3节点集群的故障场景分析

渲染错误: Mermaid 渲染失败: Parse error on line 6: …ower3] Note over A: 剩余3节点
仍有 ———————-^ Expecting 'SEMI', 'NEWLINE', 'EOF', 'AMP', 'START_LINK', 'LINK', 'LINK_ID', got 'NODE_STRING'

五、Observer节点的特殊处理

5.1 Observer宕机的影响

Observer节点不参与投票，因此它的宕机对集群几乎无影响：

#mermaid-svg-4aOVziLRQQpDH4iZ{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-4aOVziLRQQpDH4iZ .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-4aOVziLRQQpDH4iZ .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-4aOVziLRQQpDH4iZ .error-icon{fill:#552222;}#mermaid-svg-4aOVziLRQQpDH4iZ .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-4aOVziLRQQpDH4iZ .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-4aOVziLRQQpDH4iZ .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-4aOVziLRQQpDH4iZ .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-4aOVziLRQQpDH4iZ .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-4aOVziLRQQpDH4iZ .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-4aOVziLRQQpDH4iZ .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-4aOVziLRQQpDH4iZ .marker{fill:#333333;stroke:#333333;}#mermaid-svg-4aOVziLRQQpDH4iZ .marker.cross{stroke:#333333;}#mermaid-svg-4aOVziLRQQpDH4iZ svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-4aOVziLRQQpDH4iZ p{margin:0;}#mermaid-svg-4aOVziLRQQpDH4iZ .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-4aOVziLRQQpDH4iZ .cluster-label text{fill:#333;}#mermaid-svg-4aOVziLRQQpDH4iZ .cluster-label span{color:#333;}#mermaid-svg-4aOVziLRQQpDH4iZ .cluster-label span p{background-color:transparent;}#mermaid-svg-4aOVziLRQQpDH4iZ .label text,#mermaid-svg-4aOVziLRQQpDH4iZ span{fill:#333;color:#333;}#mermaid-svg-4aOVziLRQQpDH4iZ .node rect,#mermaid-svg-4aOVziLRQQpDH4iZ .node circle,#mermaid-svg-4aOVziLRQQpDH4iZ .node ellipse,#mermaid-svg-4aOVziLRQQpDH4iZ .node polygon,#mermaid-svg-4aOVziLRQQpDH4iZ .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-4aOVziLRQQpDH4iZ .rough-node .label text,#mermaid-svg-4aOVziLRQQpDH4iZ .node .label text,#mermaid-svg-4aOVziLRQQpDH4iZ .image-shape .label,#mermaid-svg-4aOVziLRQQpDH4iZ .icon-shape .label{text-anchor:middle;}#mermaid-svg-4aOVziLRQQpDH4iZ .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-4aOVziLRQQpDH4iZ .rough-node .label,#mermaid-svg-4aOVziLRQQpDH4iZ .node .label,#mermaid-svg-4aOVziLRQQpDH4iZ .image-shape .label,#mermaid-svg-4aOVziLRQQpDH4iZ .icon-shape .label{text-align:center;}#mermaid-svg-4aOVziLRQQpDH4iZ .node.clickable{cursor:pointer;}#mermaid-svg-4aOVziLRQQpDH4iZ .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-4aOVziLRQQpDH4iZ .arrowheadPath{fill:#333333;}#mermaid-svg-4aOVziLRQQpDH4iZ .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-4aOVziLRQQpDH4iZ .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-4aOVziLRQQpDH4iZ .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-4aOVziLRQQpDH4iZ .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-4aOVziLRQQpDH4iZ .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-4aOVziLRQQpDH4iZ .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-4aOVziLRQQpDH4iZ .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-4aOVziLRQQpDH4iZ .cluster text{fill:#333;}#mermaid-svg-4aOVziLRQQpDH4iZ .cluster span{color:#333;}#mermaid-svg-4aOVziLRQQpDH4iZ div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-4aOVziLRQQpDH4iZ .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-4aOVziLRQQpDH4iZ rect.text{fill:none;stroke-width:0;}#mermaid-svg-4aOVziLRQQpDH4iZ .icon-shape,#mermaid-svg-4aOVziLRQQpDH4iZ .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-4aOVziLRQQpDH4iZ .icon-shape p,#mermaid-svg-4aOVziLRQQpDH4iZ .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-4aOVziLRQQpDH4iZ .icon-shape rect,#mermaid-svg-4aOVziLRQQpDH4iZ .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-4aOVziLRQQpDH4iZ .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-4aOVziLRQQpDH4iZ .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-4aOVziLRQQpDH4iZ :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

观察者集群

核心投票集群

不影响

正常同步

正常同步

Leader

Follower1

Follower2

Follower3

Observer1 宕机

Observer2

Observer3

5.2 Observer宕机的处理

// Observer宕机时，Leader的处理
public class Leader {

public void handleObserverFailure(long sid) {
// 1. 移除观察者
observers.remove(sid);

// 2. 记录日志
LOG.info("Observer " + sid + " has been removed");

// 3. 无需选举，继续服务
// Observer不参与投票，不影响过半机制
}
}

六、故障恢复与数据一致性

6.1 宕机节点重新加入

当宕机的Follower重新启动时，需要与Leader进行数据同步：

public class RejoiningFollower {

public void rejoinCluster() throws Exception {
// 1. 连接Leader
Leader leader = connectToLeader();

// 2. 发送FOLLOWERINFO，包含lastZxid
long lastZxid = getLastZxidFromLog();

// 3. Leader决定同步策略
// – DIFF: 如果只是落后少量事务
// – SNAP: 如果落后太多
// – TRUNC: 如果有多余事务（旧Leader恢复）

// 4. 执行同步
syncWithLeader();

// 5. 状态变为FOLLOWING
// 6. 开始正常服务
}
}

6.2 一致性保证

即使经过多次宕机和恢复，ZooKeeper仍能保证最终一致性：

Follower1(曾宕机)

Leader

客户端

Follower1(曾宕机)

Leader

客户端

#mermaid-svg-6EragpiMcbM00ixA{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-6EragpiMcbM00ixA .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-6EragpiMcbM00ixA .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-6EragpiMcbM00ixA .error-icon{fill:#552222;}#mermaid-svg-6EragpiMcbM00ixA .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-6EragpiMcbM00ixA .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-6EragpiMcbM00ixA .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-6EragpiMcbM00ixA .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-6EragpiMcbM00ixA .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-6EragpiMcbM00ixA .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-6EragpiMcbM00ixA .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-6EragpiMcbM00ixA .marker{fill:#333333;stroke:#333333;}#mermaid-svg-6EragpiMcbM00ixA .marker.cross{stroke:#333333;}#mermaid-svg-6EragpiMcbM00ixA svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-6EragpiMcbM00ixA p{margin:0;}#mermaid-svg-6EragpiMcbM00ixA .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-6EragpiMcbM00ixA text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-6EragpiMcbM00ixA .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-6EragpiMcbM00ixA .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-6EragpiMcbM00ixA .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-6EragpiMcbM00ixA .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-6EragpiMcbM00ixA #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-6EragpiMcbM00ixA .sequenceNumber{fill:white;}#mermaid-svg-6EragpiMcbM00ixA #sequencenumber{fill:#333;}#mermaid-svg-6EragpiMcbM00ixA #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-6EragpiMcbM00ixA .messageText{fill:#333;stroke:none;}#mermaid-svg-6EragpiMcbM00ixA .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-6EragpiMcbM00ixA .labelText,#mermaid-svg-6EragpiMcbM00ixA .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-6EragpiMcbM00ixA .loopText,#mermaid-svg-6EragpiMcbM00ixA .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-6EragpiMcbM00ixA .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-6EragpiMcbM00ixA .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-6EragpiMcbM00ixA .noteText,#mermaid-svg-6EragpiMcbM00ixA .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-6EragpiMcbM00ixA .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-6EragpiMcbM00ixA .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-6EragpiMcbM00ixA .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-6EragpiMcbM00ixA .actorPopupMenu{position:absolute;}#mermaid-svg-6EragpiMcbM00ixA .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-6EragpiMcbM00ixA .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-6EragpiMcbM00ixA .actor-man circle,#mermaid-svg-6EragpiMcbM00ixA line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-6EragpiMcbM00ixA :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

写操作 ZXID=100

同步数据(新加入)

同步完成

读请求

返回最新数据(ZXID=100)

七、最佳实践与监控

7.1 集群规划建议

7.2 监控指标

# 实时监控集群状态
echo mntr | nc localhost 2181
# 重点关注：
zk_server_state leader/follower # 节点角色
zk_peer_state # 节点间连接状态
zk_outstanding_requests # 待处理请求数
zk_num_alive_connections # 活跃连接数

# 告警阈值建议
# – 节点数低于过半要求
# – Leader频繁切换
# – 长时间存在宕机节点

八、总结

8.1 宕机处理的核心机制

8.2 宕机处理流程回顾

#mermaid-svg-aIYgVrQ0tkxSJ2yx{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-aIYgVrQ0tkxSJ2yx .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .error-icon{fill:#552222;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .marker{fill:#333333;stroke:#333333;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .marker.cross{stroke:#333333;}#mermaid-svg-aIYgVrQ0tkxSJ2yx svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-aIYgVrQ0tkxSJ2yx p{margin:0;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .cluster-label text{fill:#333;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .cluster-label span{color:#333;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .cluster-label span p{background-color:transparent;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .label text,#mermaid-svg-aIYgVrQ0tkxSJ2yx span{fill:#333;color:#333;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .node rect,#mermaid-svg-aIYgVrQ0tkxSJ2yx .node circle,#mermaid-svg-aIYgVrQ0tkxSJ2yx .node ellipse,#mermaid-svg-aIYgVrQ0tkxSJ2yx .node polygon,#mermaid-svg-aIYgVrQ0tkxSJ2yx .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .rough-node .label text,#mermaid-svg-aIYgVrQ0tkxSJ2yx .node .label text,#mermaid-svg-aIYgVrQ0tkxSJ2yx .image-shape .label,#mermaid-svg-aIYgVrQ0tkxSJ2yx .icon-shape .label{text-anchor:middle;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .rough-node .label,#mermaid-svg-aIYgVrQ0tkxSJ2yx .node .label,#mermaid-svg-aIYgVrQ0tkxSJ2yx .image-shape .label,#mermaid-svg-aIYgVrQ0tkxSJ2yx .icon-shape .label{text-align:center;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .node.clickable{cursor:pointer;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .arrowheadPath{fill:#333333;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-aIYgVrQ0tkxSJ2yx .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-aIYgVrQ0tkxSJ2yx .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-aIYgVrQ0tkxSJ2yx .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .cluster text{fill:#333;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .cluster span{color:#333;}#mermaid-svg-aIYgVrQ0tkxSJ2yx div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-aIYgVrQ0tkxSJ2yx rect.text{fill:none;stroke-width:0;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .icon-shape,#mermaid-svg-aIYgVrQ0tkxSJ2yx .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .icon-shape p,#mermaid-svg-aIYgVrQ0tkxSJ2yx .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .icon-shape rect,#mermaid-svg-aIYgVrQ0tkxSJ2yx .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-aIYgVrQ0tkxSJ2yx .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-aIYgVrQ0tkxSJ2yx .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-aIYgVrQ0tkxSJ2yx :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

宕机处理全流程

Follower

Leader

节点宕机

判断角色

Leader检测到超时

移除Follower

集群降级运行

节点恢复后重新加入

Follower检测到失联

进入LOOKING状态

Leader选举

数据同步

恢复服务

8.3 一句话总结

ZooKeeper通过心跳检测发现宕机节点，利用过半机制保证在部分节点宕机时集群仍可用，通过Leader选举处理核心节点故障，并通过数据同步确保恢复后的节点与集群状态一致，最终构建了一个高可用的分布式协调系统。