HBase宕机机制深度解析：HMaster与HRegionServer故障应对

HBase宕机机制深度解析：HMaster与HRegionServer故障应对

• • 引言
  • 一、HBase集群架构回顾
  • • 1.1 核心组件
  • 二、HRegionServer宕机处理
  • • 2.1 你的理解验证
    • 2.2 宕机检测流程
    • 2.3 日志分裂与回放
    • 2.4 恢复过程可视化
    • 2.5 恢复时间影响因素
  • 三、HMaster宕机处理
  • • 3.1 你的理解验证
    • 3.2 Master选举机制
    • 3.3 Master故障转移实现
    • 3.4 Active与Standby职责
    • 3.5 Master切换对业务的影响
  • 四、高可用配置
  • • 4.1 HBase配置
    • 4.2 启动多个HMaster
  • 五、故障演练与测试
  • • 5.1 模拟RegionServer宕机
    • 5.2 模拟Master宕机
  • 六、监控与告警
  • • 6.1 关键监控指标
    • 6.2 监控脚本示例
  • 七、总结

引言

在分布式系统中，故障是常态而非例外。HBase作为高性能分布式数据库，设计了完善的高可用机制来应对各种宕机场景。无论是HRegionServer宕机还是HMaster宕机，HBase都能保证数据不丢失和服务持续可用。本文将深入解析这两种宕机场景的处理机制。

一、HBase集群架构回顾

1.1 核心组件

#mermaid-svg-tf18R7GkQVp2ut6M{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-tf18R7GkQVp2ut6M .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-tf18R7GkQVp2ut6M .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-tf18R7GkQVp2ut6M .error-icon{fill:#552222;}#mermaid-svg-tf18R7GkQVp2ut6M .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-tf18R7GkQVp2ut6M .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-tf18R7GkQVp2ut6M .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-tf18R7GkQVp2ut6M .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-tf18R7GkQVp2ut6M .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-tf18R7GkQVp2ut6M .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-tf18R7GkQVp2ut6M .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-tf18R7GkQVp2ut6M .marker{fill:#333333;stroke:#333333;}#mermaid-svg-tf18R7GkQVp2ut6M .marker.cross{stroke:#333333;}#mermaid-svg-tf18R7GkQVp2ut6M svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-tf18R7GkQVp2ut6M p{margin:0;}#mermaid-svg-tf18R7GkQVp2ut6M .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-tf18R7GkQVp2ut6M .cluster-label text{fill:#333;}#mermaid-svg-tf18R7GkQVp2ut6M .cluster-label span{color:#333;}#mermaid-svg-tf18R7GkQVp2ut6M .cluster-label span p{background-color:transparent;}#mermaid-svg-tf18R7GkQVp2ut6M .label text,#mermaid-svg-tf18R7GkQVp2ut6M span{fill:#333;color:#333;}#mermaid-svg-tf18R7GkQVp2ut6M .node rect,#mermaid-svg-tf18R7GkQVp2ut6M .node circle,#mermaid-svg-tf18R7GkQVp2ut6M .node ellipse,#mermaid-svg-tf18R7GkQVp2ut6M .node polygon,#mermaid-svg-tf18R7GkQVp2ut6M .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-tf18R7GkQVp2ut6M .rough-node .label text,#mermaid-svg-tf18R7GkQVp2ut6M .node .label text,#mermaid-svg-tf18R7GkQVp2ut6M .image-shape .label,#mermaid-svg-tf18R7GkQVp2ut6M .icon-shape .label{text-anchor:middle;}#mermaid-svg-tf18R7GkQVp2ut6M .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-tf18R7GkQVp2ut6M .rough-node .label,#mermaid-svg-tf18R7GkQVp2ut6M .node .label,#mermaid-svg-tf18R7GkQVp2ut6M .image-shape .label,#mermaid-svg-tf18R7GkQVp2ut6M .icon-shape .label{text-align:center;}#mermaid-svg-tf18R7GkQVp2ut6M .node.clickable{cursor:pointer;}#mermaid-svg-tf18R7GkQVp2ut6M .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-tf18R7GkQVp2ut6M .arrowheadPath{fill:#333333;}#mermaid-svg-tf18R7GkQVp2ut6M .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-tf18R7GkQVp2ut6M .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-tf18R7GkQVp2ut6M .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-tf18R7GkQVp2ut6M .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-tf18R7GkQVp2ut6M .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-tf18R7GkQVp2ut6M .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-tf18R7GkQVp2ut6M .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-tf18R7GkQVp2ut6M .cluster text{fill:#333;}#mermaid-svg-tf18R7GkQVp2ut6M .cluster span{color:#333;}#mermaid-svg-tf18R7GkQVp2ut6M div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-tf18R7GkQVp2ut6M .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-tf18R7GkQVp2ut6M rect.text{fill:none;stroke-width:0;}#mermaid-svg-tf18R7GkQVp2ut6M .icon-shape,#mermaid-svg-tf18R7GkQVp2ut6M .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-tf18R7GkQVp2ut6M .icon-shape p,#mermaid-svg-tf18R7GkQVp2ut6M .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-tf18R7GkQVp2ut6M .icon-shape rect,#mermaid-svg-tf18R7GkQVp2ut6M .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-tf18R7GkQVp2ut6M .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-tf18R7GkQVp2ut6M .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-tf18R7GkQVp2ut6M :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

HBase集群

RegionServer节点

Master节点

监控

监控

监控

监控

监控

监控

管理

管理

管理

管理

RegionServer1

ZooKeeper集群

HMaster Active

HMaster Standby

RegionServer2

RegionServer3

RegionServer4

HDFS数据持久层

二、HRegionServer宕机处理

2.1 你的理解验证

“如果是HRegionServer宕机，HMaster会将其所管理的region重新分布到其他活动的RegionServer上，由于数据和日志都持久在HDFS中，该操作不会导致数据丢失。所以数据的一致性和安全性是有保障的。”

✅ 完全正确！ 这是HBase高可用性的核心机制。

2.2 宕机检测流程

HDFS

其他RegionServer

HMaster

ZooKeeper

RegionServer

HDFS

其他RegionServer

HMaster

ZooKeeper

RegionServer

#mermaid-svg-J86HnWhc59REmm98{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-J86HnWhc59REmm98 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-J86HnWhc59REmm98 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-J86HnWhc59REmm98 .error-icon{fill:#552222;}#mermaid-svg-J86HnWhc59REmm98 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-J86HnWhc59REmm98 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-J86HnWhc59REmm98 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-J86HnWhc59REmm98 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-J86HnWhc59REmm98 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-J86HnWhc59REmm98 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-J86HnWhc59REmm98 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-J86HnWhc59REmm98 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-J86HnWhc59REmm98 .marker.cross{stroke:#333333;}#mermaid-svg-J86HnWhc59REmm98 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-J86HnWhc59REmm98 p{margin:0;}#mermaid-svg-J86HnWhc59REmm98 .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-J86HnWhc59REmm98 text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-J86HnWhc59REmm98 .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-J86HnWhc59REmm98 .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-J86HnWhc59REmm98 .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-J86HnWhc59REmm98 .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-J86HnWhc59REmm98 #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-J86HnWhc59REmm98 .sequenceNumber{fill:white;}#mermaid-svg-J86HnWhc59REmm98 #sequencenumber{fill:#333;}#mermaid-svg-J86HnWhc59REmm98 #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-J86HnWhc59REmm98 .messageText{fill:#333;stroke:none;}#mermaid-svg-J86HnWhc59REmm98 .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-J86HnWhc59REmm98 .labelText,#mermaid-svg-J86HnWhc59REmm98 .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-J86HnWhc59REmm98 .loopText,#mermaid-svg-J86HnWhc59REmm98 .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-J86HnWhc59REmm98 .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-J86HnWhc59REmm98 .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-J86HnWhc59REmm98 .noteText,#mermaid-svg-J86HnWhc59REmm98 .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-J86HnWhc59REmm98 .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-J86HnWhc59REmm98 .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-J86HnWhc59REmm98 .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-J86HnWhc59REmm98 .actorPopupMenu{position:absolute;}#mermaid-svg-J86HnWhc59REmm98 .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-J86HnWhc59REmm98 .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-J86HnWhc59REmm98 .actor-man circle,#mermaid-svg-J86HnWhc59REmm98 line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-J86HnWhc59REmm98 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

心跳超时未收到

par

[并行恢复]

定期发送心跳

通知RegionServer宕机

获取宕机RS上的region列表

读取WAL日志

返回日志数据

将region分配到其他RS

分配region A

分配region B

分配region C

回放WAL恢复数据

恢复完成

更新元数据

2.3 日志分裂与回放

// WAL日志处理过程
public class WALRecovery {

public void recoverRegionServer(String deadServer) {
// 1. 获取宕机服务器的WAL日志
Path walPath = getWALPath(deadServer);

// 2. 日志分裂 – 按region拆分
Map<byte[], List<WAL.Entry>> regionLogs = splitLogByRegion(walPath);

// 3. 分配到新的RegionServer
for (Map.Entry<byte[], List<WAL.Entry>> entry : regionLogs.entrySet()) {
byte[] regionName = entry.getKey();
List<WAL.Entry> logs = entry.getValue();

// 选择合适的RegionServer
ServerName targetServer = assignRegion(regionName);

// 发送日志到目标Server进行回放
replayLogs(targetServer, regionName, logs);
}
}

private void replayLogs(ServerName server, byte[] region, List<WAL.Entry> logs) {
// 在目标RegionServer上回放日志
for (WAL.Entry log : logs) {
// 重新应用写入操作
applyWriteOperation(log);
}
}
}

2.4 恢复过程可视化

#mermaid-svg-cIVsr6r0cnXVeBYy{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-cIVsr6r0cnXVeBYy .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-cIVsr6r0cnXVeBYy .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-cIVsr6r0cnXVeBYy .error-icon{fill:#552222;}#mermaid-svg-cIVsr6r0cnXVeBYy .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-cIVsr6r0cnXVeBYy .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-cIVsr6r0cnXVeBYy .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-cIVsr6r0cnXVeBYy .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-cIVsr6r0cnXVeBYy .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-cIVsr6r0cnXVeBYy .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-cIVsr6r0cnXVeBYy .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-cIVsr6r0cnXVeBYy .marker{fill:#333333;stroke:#333333;}#mermaid-svg-cIVsr6r0cnXVeBYy .marker.cross{stroke:#333333;}#mermaid-svg-cIVsr6r0cnXVeBYy svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-cIVsr6r0cnXVeBYy p{margin:0;}#mermaid-svg-cIVsr6r0cnXVeBYy .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-cIVsr6r0cnXVeBYy .cluster-label text{fill:#333;}#mermaid-svg-cIVsr6r0cnXVeBYy .cluster-label span{color:#333;}#mermaid-svg-cIVsr6r0cnXVeBYy .cluster-label span p{background-color:transparent;}#mermaid-svg-cIVsr6r0cnXVeBYy .label text,#mermaid-svg-cIVsr6r0cnXVeBYy span{fill:#333;color:#333;}#mermaid-svg-cIVsr6r0cnXVeBYy .node rect,#mermaid-svg-cIVsr6r0cnXVeBYy .node circle,#mermaid-svg-cIVsr6r0cnXVeBYy .node ellipse,#mermaid-svg-cIVsr6r0cnXVeBYy .node polygon,#mermaid-svg-cIVsr6r0cnXVeBYy .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-cIVsr6r0cnXVeBYy .rough-node .label text,#mermaid-svg-cIVsr6r0cnXVeBYy .node .label text,#mermaid-svg-cIVsr6r0cnXVeBYy .image-shape .label,#mermaid-svg-cIVsr6r0cnXVeBYy .icon-shape .label{text-anchor:middle;}#mermaid-svg-cIVsr6r0cnXVeBYy .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-cIVsr6r0cnXVeBYy .rough-node .label,#mermaid-svg-cIVsr6r0cnXVeBYy .node .label,#mermaid-svg-cIVsr6r0cnXVeBYy .image-shape .label,#mermaid-svg-cIVsr6r0cnXVeBYy .icon-shape .label{text-align:center;}#mermaid-svg-cIVsr6r0cnXVeBYy .node.clickable{cursor:pointer;}#mermaid-svg-cIVsr6r0cnXVeBYy .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-cIVsr6r0cnXVeBYy .arrowheadPath{fill:#333333;}#mermaid-svg-cIVsr6r0cnXVeBYy .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-cIVsr6r0cnXVeBYy .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-cIVsr6r0cnXVeBYy .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-cIVsr6r0cnXVeBYy .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-cIVsr6r0cnXVeBYy .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-cIVsr6r0cnXVeBYy .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-cIVsr6r0cnXVeBYy .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-cIVsr6r0cnXVeBYy .cluster text{fill:#333;}#mermaid-svg-cIVsr6r0cnXVeBYy .cluster span{color:#333;}#mermaid-svg-cIVsr6r0cnXVeBYy div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-cIVsr6r0cnXVeBYy .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-cIVsr6r0cnXVeBYy rect.text{fill:none;stroke-width:0;}#mermaid-svg-cIVsr6r0cnXVeBYy .icon-shape,#mermaid-svg-cIVsr6r0cnXVeBYy .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-cIVsr6r0cnXVeBYy .icon-shape p,#mermaid-svg-cIVsr6r0cnXVeBYy .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-cIVsr6r0cnXVeBYy .icon-shape rect,#mermaid-svg-cIVsr6r0cnXVeBYy .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-cIVsr6r0cnXVeBYy .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-cIVsr6r0cnXVeBYy .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-cIVsr6r0cnXVeBYy :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

HMaster重新分配

RegionServer宕机前

分配Region A

分配Region B

分配Region C

恢复后

RegionServer1宕机

RegionServer1

Region A

Region B

Region C

RegionServer2

Region D

Region E

HMaster

RegionServer3

RegionServer4

2.5 恢复时间影响因素

三、HMaster宕机处理

3.1 你的理解验证

“如果是HMaster宕机，HMaster没有单点问题，HBase中可以启动多个HMaster，通过Zookeeper的Master Election机制保证总有一个Master运行。即ZooKeeper会保证总会有一个HMaster在对外提供服务。”

✅ 完全正确！ 这是HBase解决单点故障的核心设计。

3.2 Master选举机制

ZooKeeper

HMaster3

HMaster2

HMaster1

ZooKeeper

HMaster3

HMaster2

HMaster1

#mermaid-svg-K1UfnYUJSAjwBIH9{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-K1UfnYUJSAjwBIH9 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-K1UfnYUJSAjwBIH9 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-K1UfnYUJSAjwBIH9 .error-icon{fill:#552222;}#mermaid-svg-K1UfnYUJSAjwBIH9 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-K1UfnYUJSAjwBIH9 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-K1UfnYUJSAjwBIH9 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-K1UfnYUJSAjwBIH9 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-K1UfnYUJSAjwBIH9 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-K1UfnYUJSAjwBIH9 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-K1UfnYUJSAjwBIH9 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-K1UfnYUJSAjwBIH9 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-K1UfnYUJSAjwBIH9 .marker.cross{stroke:#333333;}#mermaid-svg-K1UfnYUJSAjwBIH9 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-K1UfnYUJSAjwBIH9 p{margin:0;}#mermaid-svg-K1UfnYUJSAjwBIH9 .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-K1UfnYUJSAjwBIH9 text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-K1UfnYUJSAjwBIH9 .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-K1UfnYUJSAjwBIH9 .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-K1UfnYUJSAjwBIH9 .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-K1UfnYUJSAjwBIH9 .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-K1UfnYUJSAjwBIH9 #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-K1UfnYUJSAjwBIH9 .sequenceNumber{fill:white;}#mermaid-svg-K1UfnYUJSAjwBIH9 #sequencenumber{fill:#333;}#mermaid-svg-K1UfnYUJSAjwBIH9 #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-K1UfnYUJSAjwBIH9 .messageText{fill:#333;stroke:none;}#mermaid-svg-K1UfnYUJSAjwBIH9 .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-K1UfnYUJSAjwBIH9 .labelText,#mermaid-svg-K1UfnYUJSAjwBIH9 .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-K1UfnYUJSAjwBIH9 .loopText,#mermaid-svg-K1UfnYUJSAjwBIH9 .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-K1UfnYUJSAjwBIH9 .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-K1UfnYUJSAjwBIH9 .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-K1UfnYUJSAjwBIH9 .noteText,#mermaid-svg-K1UfnYUJSAjwBIH9 .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-K1UfnYUJSAjwBIH9 .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-K1UfnYUJSAjwBIH9 .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-K1UfnYUJSAjwBIH9 .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-K1UfnYUJSAjwBIH9 .actorPopupMenu{position:absolute;}#mermaid-svg-K1UfnYUJSAjwBIH9 .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-K1UfnYUJSAjwBIH9 .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-K1UfnYUJSAjwBIH9 .actor-man circle,#mermaid-svg-K1UfnYUJSAjwBIH9 line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-K1UfnYUJSAjwBIH9 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

启动时竞争Master

Active Master工作

Standby监听/master节点

新的Active Master接管

创建/master临时节点

创建/master临时节点

创建/master临时节点

创建成功 (成为Active)

创建失败 (成为Standby)

创建失败 (成为Standby)

Active Master宕机

检测到/master节点消失

通知节点消失

通知节点消失

尝试创建/master

尝试创建/master

创建成功 (成为新Active)

3.3 Master故障转移实现

// Master选举的简化实现
public class MasterElection {

private ZooKeeper zk;
private String masterPath = "/hbase/master";
private boolean isActive = false;

public void joinElection() {
try {
// 尝试创建临时节点
zk.create(masterPath, getServerId().getBytes(),
ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.EPHEMERAL);

// 创建成功，成为Active Master
isActive = true;
becomeActiveMaster();

} catch (NodeExistsException e) {
// 节点已存在，成为Standby
isActive = false;
watchMasterNode();
}
}

private void watchMasterNode() {
try {
// 监听master节点
zk.exists(masterPath, new Watcher() {
@Override
public void process(WatchedEvent event) {
if (event.getType() == Event.EventType.NodeDeleted) {
// Active Master宕机，重新竞选
joinElection();
}
}
});
} catch (Exception e) {
e.printStackTrace();
}
}

private void becomeActiveMaster() {
// 1. 加载元数据
loadMetaData();

// 2. 处理RegionServer心跳
startHeartbeatListener();

// 3. 检查Region分配
balanceRegions();

// 4. 处理DDL请求
startMasterService();
}
}

3.4 Active与Standby职责

3.5 Master切换对业务的影响

#mermaid-svg-l3A40jlPfEo9BkxK{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-l3A40jlPfEo9BkxK .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-l3A40jlPfEo9BkxK .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-l3A40jlPfEo9BkxK .error-icon{fill:#552222;}#mermaid-svg-l3A40jlPfEo9BkxK .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-l3A40jlPfEo9BkxK .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-l3A40jlPfEo9BkxK .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-l3A40jlPfEo9BkxK .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-l3A40jlPfEo9BkxK .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-l3A40jlPfEo9BkxK .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-l3A40jlPfEo9BkxK .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-l3A40jlPfEo9BkxK .marker{fill:#333333;stroke:#333333;}#mermaid-svg-l3A40jlPfEo9BkxK .marker.cross{stroke:#333333;}#mermaid-svg-l3A40jlPfEo9BkxK svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-l3A40jlPfEo9BkxK p{margin:0;}#mermaid-svg-l3A40jlPfEo9BkxK .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-l3A40jlPfEo9BkxK .cluster-label text{fill:#333;}#mermaid-svg-l3A40jlPfEo9BkxK .cluster-label span{color:#333;}#mermaid-svg-l3A40jlPfEo9BkxK .cluster-label span p{background-color:transparent;}#mermaid-svg-l3A40jlPfEo9BkxK .label text,#mermaid-svg-l3A40jlPfEo9BkxK span{fill:#333;color:#333;}#mermaid-svg-l3A40jlPfEo9BkxK .node rect,#mermaid-svg-l3A40jlPfEo9BkxK .node circle,#mermaid-svg-l3A40jlPfEo9BkxK .node ellipse,#mermaid-svg-l3A40jlPfEo9BkxK .node polygon,#mermaid-svg-l3A40jlPfEo9BkxK .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-l3A40jlPfEo9BkxK .rough-node .label text,#mermaid-svg-l3A40jlPfEo9BkxK .node .label text,#mermaid-svg-l3A40jlPfEo9BkxK .image-shape .label,#mermaid-svg-l3A40jlPfEo9BkxK .icon-shape .label{text-anchor:middle;}#mermaid-svg-l3A40jlPfEo9BkxK .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-l3A40jlPfEo9BkxK .rough-node .label,#mermaid-svg-l3A40jlPfEo9BkxK .node .label,#mermaid-svg-l3A40jlPfEo9BkxK .image-shape .label,#mermaid-svg-l3A40jlPfEo9BkxK .icon-shape .label{text-align:center;}#mermaid-svg-l3A40jlPfEo9BkxK .node.clickable{cursor:pointer;}#mermaid-svg-l3A40jlPfEo9BkxK .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-l3A40jlPfEo9BkxK .arrowheadPath{fill:#333333;}#mermaid-svg-l3A40jlPfEo9BkxK .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-l3A40jlPfEo9BkxK .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-l3A40jlPfEo9BkxK .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-l3A40jlPfEo9BkxK .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-l3A40jlPfEo9BkxK .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-l3A40jlPfEo9BkxK .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-l3A40jlPfEo9BkxK .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-l3A40jlPfEo9BkxK .cluster text{fill:#333;}#mermaid-svg-l3A40jlPfEo9BkxK .cluster span{color:#333;}#mermaid-svg-l3A40jlPfEo9BkxK div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-l3A40jlPfEo9BkxK .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-l3A40jlPfEo9BkxK rect.text{fill:none;stroke-width:0;}#mermaid-svg-l3A40jlPfEo9BkxK .icon-shape,#mermaid-svg-l3A40jlPfEo9BkxK .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-l3A40jlPfEo9BkxK .icon-shape p,#mermaid-svg-l3A40jlPfEo9BkxK .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-l3A40jlPfEo9BkxK .icon-shape rect,#mermaid-svg-l3A40jlPfEo9BkxK .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-l3A40jlPfEo9BkxK .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-l3A40jlPfEo9BkxK .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-l3A40jlPfEo9BkxK :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Master切换时间线

宕机

Active运行

ZooKeeper检测约30秒

Standby竞选毫秒级

新Active加载元数据秒级

服务恢复

四、高可用配置

4.1 HBase配置

<!– hbase-site.xml 高可用配置 –>
<configuration>
<!– ZooKeeper集群地址 –>
<property>
<name>hbase.zookeeper.quorum</name>
<value>zk1:2181,zk2:2181,zk3:2181</value>
</property>

<!– 开启Master高可用 –>
<property>
<name>hbase.master.port</name>
<value>16000</value>
</property>

<!– 备份Master数量 –>
<property>
<name>hbase.master.backup</name>
<value>2</value>
</property>

<!– RegionServer故障恢复相关 –>
<property>
<name>hbase.regionserver.wal.max.splitters</name>
<value>2</value>
</property>

<property>
<name>hbase.regionserver.wal.max.splits</name>
<value>2</value>
</property>
</configuration>

4.2 启动多个HMaster

# 启动Active Master
$HBASE_HOME/bin/hbase-daemon.sh start master

# 启动Standby Master（在同一集群的其他节点）
$HBASE_HOME/bin/hbase-daemon.sh start master

# 查看Master状态
$HBASE_HOME/bin/hbase hbck
# 或通过Web UI
# http://active-master:16010
# http://standby-master:16010 (会重定向到Active)

五、故障演练与测试

5.1 模拟RegionServer宕机

# 1. 查看当前RegionServer
$HBASE_HOME/bin/hbase shell
> status 'detailed'

# 2. 模拟宕机（在RegionServer节点执行）
kill -9 <RegionServer PID>

# 3. 观察恢复过程
$HBASE_HOME/bin/hbase hbck
# 查看HBase日志
tail -f $HBASE_HOME/logs/hbase-hbase-master-*.log

5.2 模拟Master宕机

# 1. 查看当前Active Master
$HBASE_HOME/bin/hbase shell
> status 'master'

# 2. 模拟Active Master宕机
kill -9 <Active Master PID>

# 3. 观察Standby接管
tail -f $HBASE_HOME/logs/hbase-hbase-master-*.log
# 会看到Standby成为Active的日志

# 4. 验证服务是否正常
$HBASE_HOME/bin/hbase shell
> list # 应该能正常执行

六、监控与告警

6.1 关键监控指标

6.2 监控脚本示例

#!/bin/bash
# check_hbase_ha.sh

# 检查Master数量
MASTER_COUNT=$(jps | grep HMaster | wc -l)
if [ $MASTER_COUNT -lt 2 ]; then
echo "WARNING: Only $MASTER_COUNT HMaster running"
fi

# 检查RegionServer数量
RS_COUNT=$(jps | grep HRegionServer | wc -l)
if [ $RS_COUNT -lt 3 ]; then
echo "CRITICAL: Only $RS_COUNT RegionServer running"
fi

# 检查ZooKeeper连接
echo "stat" | nc zk1 2181 | grep Mode

七、总结

核心要点：

一句话总结：HBase通过ZooKeeper的协调机制，结合HDFS的持久化能力，实现了对RegionServer和Master宕机的自动恢复，确保了系统的高可用性和数据一致性。