Hadoop任务容错机制深度解析：重试机制与故障恢复实战

Hadoop任务容错机制深度解析：重试机制与故障恢复实战

• • 引言：容错——分布式系统的生存法则
  • 一、Hadoop容错体系全景
  • • 1.1 容错的多层次防护
    • 1.2 容错设计的核心思想
  • 二、任务级容错：重试机制深度解析
  • • 2.1 任务失败的原因分类
    • 2.2 重试机制的工作流程
    • 2.3 核心重试参数
    • 2.4 重试机制的内部实现
    • 2.5 指数退避重试策略
  • 三、推测执行：加速任务完成的容错机制
  • • 3.1 什么是推测执行？
    • 3.2 推测执行的工作机制
    • 3.3 推测执行配置
    • 3.4 推测执行的适用场景
  • 四、节点级容错：任务迁移与数据恢复
  • • 4.1 节点失效检测
    • 4.2 节点失效后的任务处理
    • 4.3 数据本地性失效的处理
  • 五、应用级容错：ApplicationMaster恢复
  • • 5.1 AM失效的影响
    • 5.2 AM恢复机制
    • 5.3 AM恢复配置
  • 六、集群级容错：HA架构
  • • 6.1 NameNode HA
    • 6.2 ResourceManager HA
  • 七、容错机制的监控与调优
  • • 7.1 关键监控指标
    • 7.2 常见问题排查
    • 7.3 调优建议
  • 八、总结：容错机制的最佳实践
  • • 8.1 容错体系核心要点
    • 8.2 黄金法则
    • 8.3 最终建议

引言：容错——分布式系统的生存法则

在分布式系统中，节点故障、网络波动、任务超时是常态而非异常。Hadoop作为大规模数据处理平台，设计了一套多层次的容错机制，确保即使部分组件失效，整个作业仍能成功完成。而重试机制正是这套容错体系中最核心、最常用的手段。

本文将深入剖析Hadoop的任务级容错原理，从故障检测到自动恢复，全面解析重试机制的实现细节。

#mermaid-svg-ZNUrd79IcuZj9Apv{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ZNUrd79IcuZj9Apv .error-icon{fill:#552222;}#mermaid-svg-ZNUrd79IcuZj9Apv .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ZNUrd79IcuZj9Apv .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ZNUrd79IcuZj9Apv .marker.cross{stroke:#333333;}#mermaid-svg-ZNUrd79IcuZj9Apv svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ZNUrd79IcuZj9Apv p{margin:0;}#mermaid-svg-ZNUrd79IcuZj9Apv .edge{stroke-width:3;}#mermaid-svg-ZNUrd79IcuZj9Apv .section–1 rect,#mermaid-svg-ZNUrd79IcuZj9Apv .section–1 path,#mermaid-svg-ZNUrd79IcuZj9Apv .section–1 circle,#mermaid-svg-ZNUrd79IcuZj9Apv .section–1 polygon,#mermaid-svg-ZNUrd79IcuZj9Apv .section–1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .section–1 text{fill:#ffffff;}#mermaid-svg-ZNUrd79IcuZj9Apv .node-icon–1{font-size:40px;color:#ffffff;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-edge–1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-depth–1{stroke-width:17;}#mermaid-svg-ZNUrd79IcuZj9Apv .section–1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled circle,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:lightgray;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:#efefef;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-0 rect,#mermaid-svg-ZNUrd79IcuZj9Apv .section-0 path,#mermaid-svg-ZNUrd79IcuZj9Apv .section-0 circle,#mermaid-svg-ZNUrd79IcuZj9Apv .section-0 polygon,#mermaid-svg-ZNUrd79IcuZj9Apv .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-ZNUrd79IcuZj9Apv .section-0 text{fill:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .node-icon-0{font-size:40px;color:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-depth-0{stroke-width:14;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled circle,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:lightgray;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:#efefef;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-1 rect,#mermaid-svg-ZNUrd79IcuZj9Apv .section-1 path,#mermaid-svg-ZNUrd79IcuZj9Apv .section-1 circle,#mermaid-svg-ZNUrd79IcuZj9Apv .section-1 polygon,#mermaid-svg-ZNUrd79IcuZj9Apv .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .section-1 text{fill:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .node-icon-1{font-size:40px;color:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-depth-1{stroke-width:11;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled circle,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:lightgray;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:#efefef;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-2 rect,#mermaid-svg-ZNUrd79IcuZj9Apv .section-2 path,#mermaid-svg-ZNUrd79IcuZj9Apv .section-2 circle,#mermaid-svg-ZNUrd79IcuZj9Apv .section-2 polygon,#mermaid-svg-ZNUrd79IcuZj9Apv .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .section-2 text{fill:#ffffff;}#mermaid-svg-ZNUrd79IcuZj9Apv .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-depth-2{stroke-width:8;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled circle,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:lightgray;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:#efefef;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-3 rect,#mermaid-svg-ZNUrd79IcuZj9Apv .section-3 path,#mermaid-svg-ZNUrd79IcuZj9Apv .section-3 circle,#mermaid-svg-ZNUrd79IcuZj9Apv .section-3 polygon,#mermaid-svg-ZNUrd79IcuZj9Apv .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .section-3 text{fill:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .node-icon-3{font-size:40px;color:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-depth-3{stroke-width:5;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled circle,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:lightgray;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:#efefef;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-4 rect,#mermaid-svg-ZNUrd79IcuZj9Apv .section-4 path,#mermaid-svg-ZNUrd79IcuZj9Apv .section-4 circle,#mermaid-svg-ZNUrd79IcuZj9Apv .section-4 polygon,#mermaid-svg-ZNUrd79IcuZj9Apv .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .section-4 text{fill:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .node-icon-4{font-size:40px;color:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-depth-4{stroke-width:2;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled circle,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:lightgray;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:#efefef;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-5 rect,#mermaid-svg-ZNUrd79IcuZj9Apv .section-5 path,#mermaid-svg-ZNUrd79IcuZj9Apv .section-5 circle,#mermaid-svg-ZNUrd79IcuZj9Apv .section-5 polygon,#mermaid-svg-ZNUrd79IcuZj9Apv .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .section-5 text{fill:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .node-icon-5{font-size:40px;color:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-depth-5{stroke-width:-1;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled circle,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:lightgray;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:#efefef;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-6 rect,#mermaid-svg-ZNUrd79IcuZj9Apv .section-6 path,#mermaid-svg-ZNUrd79IcuZj9Apv .section-6 circle,#mermaid-svg-ZNUrd79IcuZj9Apv .section-6 polygon,#mermaid-svg-ZNUrd79IcuZj9Apv .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .section-6 text{fill:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .node-icon-6{font-size:40px;color:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-depth-6{stroke-width:-4;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled circle,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:lightgray;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:#efefef;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-7 rect,#mermaid-svg-ZNUrd79IcuZj9Apv .section-7 path,#mermaid-svg-ZNUrd79IcuZj9Apv .section-7 circle,#mermaid-svg-ZNUrd79IcuZj9Apv .section-7 polygon,#mermaid-svg-ZNUrd79IcuZj9Apv .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .section-7 text{fill:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .node-icon-7{font-size:40px;color:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-depth-7{stroke-width:-7;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled circle,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:lightgray;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:#efefef;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-8 rect,#mermaid-svg-ZNUrd79IcuZj9Apv .section-8 path,#mermaid-svg-ZNUrd79IcuZj9Apv .section-8 circle,#mermaid-svg-ZNUrd79IcuZj9Apv .section-8 polygon,#mermaid-svg-ZNUrd79IcuZj9Apv .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .section-8 text{fill:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .node-icon-8{font-size:40px;color:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-depth-8{stroke-width:-10;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled circle,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:lightgray;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:#efefef;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-9 rect,#mermaid-svg-ZNUrd79IcuZj9Apv .section-9 path,#mermaid-svg-ZNUrd79IcuZj9Apv .section-9 circle,#mermaid-svg-ZNUrd79IcuZj9Apv .section-9 polygon,#mermaid-svg-ZNUrd79IcuZj9Apv .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .section-9 text{fill:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .node-icon-9{font-size:40px;color:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-depth-9{stroke-width:-13;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled circle,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:lightgray;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:#efefef;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-10 rect,#mermaid-svg-ZNUrd79IcuZj9Apv .section-10 path,#mermaid-svg-ZNUrd79IcuZj9Apv .section-10 circle,#mermaid-svg-ZNUrd79IcuZj9Apv .section-10 polygon,#mermaid-svg-ZNUrd79IcuZj9Apv .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .section-10 text{fill:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .node-icon-10{font-size:40px;color:black;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .edge-depth-10{stroke-width:-16;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled circle,#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:lightgray;}#mermaid-svg-ZNUrd79IcuZj9Apv .disabled text{fill:#efefef;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-root rect,#mermaid-svg-ZNUrd79IcuZj9Apv .section-root path,#mermaid-svg-ZNUrd79IcuZj9Apv .section-root circle,#mermaid-svg-ZNUrd79IcuZj9Apv .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-ZNUrd79IcuZj9Apv .section-root text{fill:#ffffff;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-root span{color:#ffffff;}#mermaid-svg-ZNUrd79IcuZj9Apv .section-2 span{color:#ffffff;}#mermaid-svg-ZNUrd79IcuZj9Apv .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-ZNUrd79IcuZj9Apv .edge{fill:none;}#mermaid-svg-ZNUrd79IcuZj9Apv .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-ZNUrd79IcuZj9Apv :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Hadoop容错机制

故障检测

心跳机制

超时判定

节点健康检查

重试策略

Map任务重试

Reduce任务重试

ApplicationMaster重试

指数退避算法

资源隔离

Container隔离

任务推测执行

故障转移

数据容错

副本机制

数据块校验

任务输出原子性

一、Hadoop容错体系全景

1.1 容错的多层次防护

#mermaid-svg-Kj0ojG0QzvLWl2EU{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Kj0ojG0QzvLWl2EU .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Kj0ojG0QzvLWl2EU .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Kj0ojG0QzvLWl2EU .error-icon{fill:#552222;}#mermaid-svg-Kj0ojG0QzvLWl2EU .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Kj0ojG0QzvLWl2EU .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Kj0ojG0QzvLWl2EU .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Kj0ojG0QzvLWl2EU .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Kj0ojG0QzvLWl2EU .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Kj0ojG0QzvLWl2EU .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Kj0ojG0QzvLWl2EU .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Kj0ojG0QzvLWl2EU .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Kj0ojG0QzvLWl2EU .marker.cross{stroke:#333333;}#mermaid-svg-Kj0ojG0QzvLWl2EU svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Kj0ojG0QzvLWl2EU p{margin:0;}#mermaid-svg-Kj0ojG0QzvLWl2EU .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-Kj0ojG0QzvLWl2EU .cluster-label text{fill:#333;}#mermaid-svg-Kj0ojG0QzvLWl2EU .cluster-label span{color:#333;}#mermaid-svg-Kj0ojG0QzvLWl2EU .cluster-label span p{background-color:transparent;}#mermaid-svg-Kj0ojG0QzvLWl2EU .label text,#mermaid-svg-Kj0ojG0QzvLWl2EU span{fill:#333;color:#333;}#mermaid-svg-Kj0ojG0QzvLWl2EU .node rect,#mermaid-svg-Kj0ojG0QzvLWl2EU .node circle,#mermaid-svg-Kj0ojG0QzvLWl2EU .node ellipse,#mermaid-svg-Kj0ojG0QzvLWl2EU .node polygon,#mermaid-svg-Kj0ojG0QzvLWl2EU .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Kj0ojG0QzvLWl2EU .rough-node .label text,#mermaid-svg-Kj0ojG0QzvLWl2EU .node .label text,#mermaid-svg-Kj0ojG0QzvLWl2EU .image-shape .label,#mermaid-svg-Kj0ojG0QzvLWl2EU .icon-shape .label{text-anchor:middle;}#mermaid-svg-Kj0ojG0QzvLWl2EU .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Kj0ojG0QzvLWl2EU .rough-node .label,#mermaid-svg-Kj0ojG0QzvLWl2EU .node .label,#mermaid-svg-Kj0ojG0QzvLWl2EU .image-shape .label,#mermaid-svg-Kj0ojG0QzvLWl2EU .icon-shape .label{text-align:center;}#mermaid-svg-Kj0ojG0QzvLWl2EU .node.clickable{cursor:pointer;}#mermaid-svg-Kj0ojG0QzvLWl2EU .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Kj0ojG0QzvLWl2EU .arrowheadPath{fill:#333333;}#mermaid-svg-Kj0ojG0QzvLWl2EU .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Kj0ojG0QzvLWl2EU .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Kj0ojG0QzvLWl2EU .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Kj0ojG0QzvLWl2EU .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Kj0ojG0QzvLWl2EU .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Kj0ojG0QzvLWl2EU .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Kj0ojG0QzvLWl2EU .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Kj0ojG0QzvLWl2EU .cluster text{fill:#333;}#mermaid-svg-Kj0ojG0QzvLWl2EU .cluster span{color:#333;}#mermaid-svg-Kj0ojG0QzvLWl2EU div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Kj0ojG0QzvLWl2EU .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Kj0ojG0QzvLWl2EU rect.text{fill:none;stroke-width:0;}#mermaid-svg-Kj0ojG0QzvLWl2EU .icon-shape,#mermaid-svg-Kj0ojG0QzvLWl2EU .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Kj0ojG0QzvLWl2EU .icon-shape p,#mermaid-svg-Kj0ojG0QzvLWl2EU .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Kj0ojG0QzvLWl2EU .icon-shape rect,#mermaid-svg-Kj0ojG0QzvLWl2EU .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Kj0ojG0QzvLWl2EU .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Kj0ojG0QzvLWl2EU .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Kj0ojG0QzvLWl2EU :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

集群级容错

ResourceManager失效

NameNode切换

NameNode失效

应用级容错

ApplicationMaster失效

重新启动AM

从ZK恢复状态

节点级容错

节点失效

标记离线

任务迁移

数据复制

任务级容错

其他节点

任务失败

重试机制

1.2 容错设计的核心思想

Hadoop的容错设计遵循三大原则：

二、任务级容错：重试机制深度解析

2.1 任务失败的原因分类

MapReduce任务可能因多种原因失败：

2.2 重试机制的工作流程

ResourceManager

ApplicationMaster

NodeManager

Map/Reduce任务

ResourceManager

ApplicationMaster

NodeManager

Map/Reduce任务

#mermaid-svg-cM1SwudAKGcEg8vM{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-cM1SwudAKGcEg8vM .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-cM1SwudAKGcEg8vM .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-cM1SwudAKGcEg8vM .error-icon{fill:#552222;}#mermaid-svg-cM1SwudAKGcEg8vM .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-cM1SwudAKGcEg8vM .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-cM1SwudAKGcEg8vM .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-cM1SwudAKGcEg8vM .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-cM1SwudAKGcEg8vM .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-cM1SwudAKGcEg8vM .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-cM1SwudAKGcEg8vM .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-cM1SwudAKGcEg8vM .marker{fill:#333333;stroke:#333333;}#mermaid-svg-cM1SwudAKGcEg8vM .marker.cross{stroke:#333333;}#mermaid-svg-cM1SwudAKGcEg8vM svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-cM1SwudAKGcEg8vM p{margin:0;}#mermaid-svg-cM1SwudAKGcEg8vM .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-cM1SwudAKGcEg8vM text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-cM1SwudAKGcEg8vM .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-cM1SwudAKGcEg8vM .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-cM1SwudAKGcEg8vM .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-cM1SwudAKGcEg8vM .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-cM1SwudAKGcEg8vM #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-cM1SwudAKGcEg8vM .sequenceNumber{fill:white;}#mermaid-svg-cM1SwudAKGcEg8vM #sequencenumber{fill:#333;}#mermaid-svg-cM1SwudAKGcEg8vM #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-cM1SwudAKGcEg8vM .messageText{fill:#333;stroke:none;}#mermaid-svg-cM1SwudAKGcEg8vM .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-cM1SwudAKGcEg8vM .labelText,#mermaid-svg-cM1SwudAKGcEg8vM .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-cM1SwudAKGcEg8vM .loopText,#mermaid-svg-cM1SwudAKGcEg8vM .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-cM1SwudAKGcEg8vM .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-cM1SwudAKGcEg8vM .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-cM1SwudAKGcEg8vM .noteText,#mermaid-svg-cM1SwudAKGcEg8vM .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-cM1SwudAKGcEg8vM .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-cM1SwudAKGcEg8vM .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-cM1SwudAKGcEg8vM .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-cM1SwudAKGcEg8vM .actorPopupMenu{position:absolute;}#mermaid-svg-cM1SwudAKGcEg8vM .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-cM1SwudAKGcEg8vM .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-cM1SwudAKGcEg8vM .actor-man circle,#mermaid-svg-cM1SwudAKGcEg8vM line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-cM1SwudAKGcEg8vM :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

alt

[可重试]

[不可重试]

loop

[重试判断]

运行中

任务崩溃

通知任务失败

记录失败次数

失败次数 < 最大重试?

重新申请资源

分配新容器

在新节点启动任务

标记任务失败

检查作业整体状态

2.3 核心重试参数

<!– mapred-site.xml 重试配置 –>
<configuration>
<!– Map任务最大尝试次数 –>
<property>
<name>mapreduce.map.maxattempts</name>
<value>4</value>
<description>Map任务最大尝试次数，默认4次</description>
</property>

<!– Reduce任务最大尝试次数 –>
<property>
<name>mapreduce.reduce.maxattempts</name>
<value>4</value>
<description>Reduce任务最大尝试次数，默认4次</description>
</property>

<!– 任务失败百分比阈值 –>
<property>
<name>mapreduce.job.map.failures.percent</name>
<value>0</value>
<description>Map任务失败百分比超过此值时作业失败，0表示任意失败即失败</description>
</property>

<!– 任务超时时间 –>
<property>
<name>mapreduce.task.timeout</name>
<value>600000</value>
<description>任务无响应超时时间（毫秒），默认600000（10分钟）</description>
</property>

<!– 最大任务失败数 –>
<property>
<name>mapreduce.max.map.failures.percent</name>
<value>1</value>
<description>允许的Map任务失败百分比，超过此值时作业失败</description>
</property>
</configuration>

2.4 重试机制的内部实现

// ApplicationMaster中任务失败处理的核心逻辑
public class TaskAttemptListenerImpl {

private static final int MAX_MAP_ATTEMPTS = 4;
private static final int MAX_REDUCE_ATTEMPTS = 4;

public void taskFailed(TaskAttemptId attemptId, String diagnostics) {
TaskAttempt taskAttempt = getTaskAttempt(attemptId);
int attemptCount = taskAttempt.getAttemptCount();

// 判断是否超过最大重试次数
int maxAttempts = taskAttempt.isMapTask() ?
MAX_MAP_ATTEMPTS : MAX_REDUCE_ATTEMPTS;

if (attemptCount < maxAttempts) {
// 可以重试：调度新尝试
LOG.info("Task " + attemptId + " failed, retrying. Attempt " +
(attemptCount + 1) + " of " + maxAttempts);

// 使用指数退避策略计算重试延迟
long delay = computeRetryDelay(attemptCount);

// 调度重试
scheduleRetry(taskAttempt, delay);
} else {
// 超过最大重试次数，标记为永久失败
LOG.error("Task " + attemptId + " failed permanently after " +
maxAttempts + " attempts");
markTaskAsFailed(taskAttempt);

// 检查作业整体失败阈值
checkJobFailureThreshold();
}
}

private long computeRetryDelay(int attemptCount) {
// 指数退避算法：延迟时间随着尝试次数指数增长
// 第一次重试：30秒
// 第二次重试：60秒
// 第三次重试：120秒
long baseDelay = 30000; // 30秒
return baseDelay * (long) Math.pow(2, attemptCount – 1);
}
}

2.5 指数退避重试策略

Hadoop采用指数退避算法来避免重试风暴：

这种策略的优势：

• 临时问题（如网络抖动）可快速重试解决
• 严重故障（如节点宕机）避免频繁无效重试
• 减少集群资源浪费

三、推测执行：加速任务完成的容错机制

3.1 什么是推测执行？

推测执行是一种"预防性容错"机制：当某个任务运行速度明显慢于其他任务时，系统会在另一个节点上启动一个备份任务，哪个任务先完成就用哪个的结果。

#mermaid-svg-kAwN3J01XBbcqw3V{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-kAwN3J01XBbcqw3V .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-kAwN3J01XBbcqw3V .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-kAwN3J01XBbcqw3V .error-icon{fill:#552222;}#mermaid-svg-kAwN3J01XBbcqw3V .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-kAwN3J01XBbcqw3V .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-kAwN3J01XBbcqw3V .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-kAwN3J01XBbcqw3V .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-kAwN3J01XBbcqw3V .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-kAwN3J01XBbcqw3V .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-kAwN3J01XBbcqw3V .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-kAwN3J01XBbcqw3V .marker{fill:#333333;stroke:#333333;}#mermaid-svg-kAwN3J01XBbcqw3V .marker.cross{stroke:#333333;}#mermaid-svg-kAwN3J01XBbcqw3V svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-kAwN3J01XBbcqw3V p{margin:0;}#mermaid-svg-kAwN3J01XBbcqw3V .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-kAwN3J01XBbcqw3V .cluster-label text{fill:#333;}#mermaid-svg-kAwN3J01XBbcqw3V .cluster-label span{color:#333;}#mermaid-svg-kAwN3J01XBbcqw3V .cluster-label span p{background-color:transparent;}#mermaid-svg-kAwN3J01XBbcqw3V .label text,#mermaid-svg-kAwN3J01XBbcqw3V span{fill:#333;color:#333;}#mermaid-svg-kAwN3J01XBbcqw3V .node rect,#mermaid-svg-kAwN3J01XBbcqw3V .node circle,#mermaid-svg-kAwN3J01XBbcqw3V .node ellipse,#mermaid-svg-kAwN3J01XBbcqw3V .node polygon,#mermaid-svg-kAwN3J01XBbcqw3V .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-kAwN3J01XBbcqw3V .rough-node .label text,#mermaid-svg-kAwN3J01XBbcqw3V .node .label text,#mermaid-svg-kAwN3J01XBbcqw3V .image-shape .label,#mermaid-svg-kAwN3J01XBbcqw3V .icon-shape .label{text-anchor:middle;}#mermaid-svg-kAwN3J01XBbcqw3V .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-kAwN3J01XBbcqw3V .rough-node .label,#mermaid-svg-kAwN3J01XBbcqw3V .node .label,#mermaid-svg-kAwN3J01XBbcqw3V .image-shape .label,#mermaid-svg-kAwN3J01XBbcqw3V .icon-shape .label{text-align:center;}#mermaid-svg-kAwN3J01XBbcqw3V .node.clickable{cursor:pointer;}#mermaid-svg-kAwN3J01XBbcqw3V .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-kAwN3J01XBbcqw3V .arrowheadPath{fill:#333333;}#mermaid-svg-kAwN3J01XBbcqw3V .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-kAwN3J01XBbcqw3V .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-kAwN3J01XBbcqw3V .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-kAwN3J01XBbcqw3V .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-kAwN3J01XBbcqw3V .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-kAwN3J01XBbcqw3V .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-kAwN3J01XBbcqw3V .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-kAwN3J01XBbcqw3V .cluster text{fill:#333;}#mermaid-svg-kAwN3J01XBbcqw3V .cluster span{color:#333;}#mermaid-svg-kAwN3J01XBbcqw3V div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-kAwN3J01XBbcqw3V .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-kAwN3J01XBbcqw3V rect.text{fill:none;stroke-width:0;}#mermaid-svg-kAwN3J01XBbcqw3V .icon-shape,#mermaid-svg-kAwN3J01XBbcqw3V .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-kAwN3J01XBbcqw3V .icon-shape p,#mermaid-svg-kAwN3J01XBbcqw3V .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-kAwN3J01XBbcqw3V .icon-shape rect,#mermaid-svg-kAwN3J01XBbcqw3V .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-kAwN3J01XBbcqw3V .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-kAwN3J01XBbcqw3V .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-kAwN3J01XBbcqw3V :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

推测执行启动

常规执行

完成

完成

完成

还在运行

更快完成

原任务

Task1

完成

Task2

Task3

Task4 Slow

等待中…

启动备份任务

Task4备份

终止原任务

3.2 推测执行的工作机制

// 推测执行判定逻辑
public class SpeculativeExecutionDecider {

private static final double SLOW_TASK_THRESHOLD = 1.5;

public boolean shouldSpeculate(TaskInProgress task,
double avgProgress,
double taskProgress) {
// 判断任务是否过慢
boolean isSlow = taskProgress < avgProgress * 0.7;

// 判断是否有足够时间启动备份
boolean hasTime = task.getRunningTime() > 60000; // 已运行>1分钟

// 判断是否已有备份任务
boolean noBackup = !task.hasSpeculativeTask();

return isSlow && hasTime && noBackup;
}
}

3.3 推测执行配置

<!– mapred-site.xml 推测执行配置 –>
<property>
<name>mapreduce.map.speculative</name>
<value>true</value>
<description>是否启用Map推测执行，默认true</description>
</property>

<property>
<name>mapreduce.reduce.speculative</name>
<value>true</value>
<description>是否启用Reduce推测执行，默认true</description>
</property>

<!– 推测执行高级参数 –>
<property>
<name>mapreduce.job.speculative.speculative-cap-running-tasks</name>
<value>0.1</value>
<description>可启动备份任务的任务比例上限</description>
</property>

<property>
<name>mapreduce.job.speculative.speculative-cap-total-tasks</name>
<value>0.01</value>
<description>备份任务占总任务数比例上限</description>
</property>

<property>
<name>mapreduce.job.speculative.minimum-allowed-tasks</name>
<value>10</value>
<description>启动推测执行所需的最小任务数</description>
</property>

3.4 推测执行的适用场景

四、节点级容错：任务迁移与数据恢复

4.1 节点失效检测

YARN通过心跳机制检测节点故障：

// ResourceManager中节点失效检测
public class NodeHealthChecker {

private static final long NODE_EXPIRY_INTERVAL = 600000; // 10分钟

public void checkNodeHealth() {
long currentTime = System.currentTimeMillis();

for (NodeId nodeId : activeNodes) {
NodeStatus status = nodeStatusMap.get(nodeId);
long lastHeartbeat = status.getLastHeartbeatTime();

if (currentTime – lastHeartbeat > NODE_EXPIRY_INTERVAL) {
// 节点失效，标记为死亡
markNodeDead(nodeId);

// 通知ApplicationMaster处理该节点上的任务
notifyAMSOfNodeFailure(nodeId);

// 在HDFS层面触发数据复制
hdfsNodeFailureHandler(nodeId);
}
}
}
}

4.2 节点失效后的任务处理

#mermaid-svg-v9QPE2TWGlwlbsEL{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-v9QPE2TWGlwlbsEL .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-v9QPE2TWGlwlbsEL .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-v9QPE2TWGlwlbsEL .error-icon{fill:#552222;}#mermaid-svg-v9QPE2TWGlwlbsEL .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-v9QPE2TWGlwlbsEL .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-v9QPE2TWGlwlbsEL .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-v9QPE2TWGlwlbsEL .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-v9QPE2TWGlwlbsEL .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-v9QPE2TWGlwlbsEL .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-v9QPE2TWGlwlbsEL .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-v9QPE2TWGlwlbsEL .marker{fill:#333333;stroke:#333333;}#mermaid-svg-v9QPE2TWGlwlbsEL .marker.cross{stroke:#333333;}#mermaid-svg-v9QPE2TWGlwlbsEL svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-v9QPE2TWGlwlbsEL p{margin:0;}#mermaid-svg-v9QPE2TWGlwlbsEL .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-v9QPE2TWGlwlbsEL .cluster-label text{fill:#333;}#mermaid-svg-v9QPE2TWGlwlbsEL .cluster-label span{color:#333;}#mermaid-svg-v9QPE2TWGlwlbsEL .cluster-label span p{background-color:transparent;}#mermaid-svg-v9QPE2TWGlwlbsEL .label text,#mermaid-svg-v9QPE2TWGlwlbsEL span{fill:#333;color:#333;}#mermaid-svg-v9QPE2TWGlwlbsEL .node rect,#mermaid-svg-v9QPE2TWGlwlbsEL .node circle,#mermaid-svg-v9QPE2TWGlwlbsEL .node ellipse,#mermaid-svg-v9QPE2TWGlwlbsEL .node polygon,#mermaid-svg-v9QPE2TWGlwlbsEL .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-v9QPE2TWGlwlbsEL .rough-node .label text,#mermaid-svg-v9QPE2TWGlwlbsEL .node .label text,#mermaid-svg-v9QPE2TWGlwlbsEL .image-shape .label,#mermaid-svg-v9QPE2TWGlwlbsEL .icon-shape .label{text-anchor:middle;}#mermaid-svg-v9QPE2TWGlwlbsEL .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-v9QPE2TWGlwlbsEL .rough-node .label,#mermaid-svg-v9QPE2TWGlwlbsEL .node .label,#mermaid-svg-v9QPE2TWGlwlbsEL .image-shape .label,#mermaid-svg-v9QPE2TWGlwlbsEL .icon-shape .label{text-align:center;}#mermaid-svg-v9QPE2TWGlwlbsEL .node.clickable{cursor:pointer;}#mermaid-svg-v9QPE2TWGlwlbsEL .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-v9QPE2TWGlwlbsEL .arrowheadPath{fill:#333333;}#mermaid-svg-v9QPE2TWGlwlbsEL .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-v9QPE2TWGlwlbsEL .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-v9QPE2TWGlwlbsEL .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-v9QPE2TWGlwlbsEL .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-v9QPE2TWGlwlbsEL .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-v9QPE2TWGlwlbsEL .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-v9QPE2TWGlwlbsEL .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-v9QPE2TWGlwlbsEL .cluster text{fill:#333;}#mermaid-svg-v9QPE2TWGlwlbsEL .cluster span{color:#333;}#mermaid-svg-v9QPE2TWGlwlbsEL div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-v9QPE2TWGlwlbsEL .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-v9QPE2TWGlwlbsEL rect.text{fill:none;stroke-width:0;}#mermaid-svg-v9QPE2TWGlwlbsEL .icon-shape,#mermaid-svg-v9QPE2TWGlwlbsEL .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-v9QPE2TWGlwlbsEL .icon-shape p,#mermaid-svg-v9QPE2TWGlwlbsEL .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-v9QPE2TWGlwlbsEL .icon-shape rect,#mermaid-svg-v9QPE2TWGlwlbsEL .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-v9QPE2TWGlwlbsEL .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-v9QPE2TWGlwlbsEL .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-v9QPE2TWGlwlbsEL :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

ApplicationMaster处理

NodeManager心跳丢失

ResourceManager标记节点死亡

通知所有ApplicationMaster

获取受影响的容器列表

将任务状态标记为FAILED

重新申请资源

在新节点重启任务

4.3 数据本地性失效的处理

当节点失效导致数据本地性无法保证时，ApplicationMaster的处理策略：

// ApplicationMaster处理数据本地性失效
public class DataLocalityManager {

public ContainerRequest handleNodeFailure(Task task,
Set<NodeId> failedNodes) {
// 获取任务所需数据的位置
List<String> dataLocations = task.getDataLocations();

// 移除失效节点
dataLocations.removeAll(failedNodes);

if (dataLocations.isEmpty()) {
// 没有可用节点，请求任意节点（降级为跨机架）
return createAnyNodeRequest(task);
} else {
// 仍有可用节点，优先请求
return createLocalRequest(task, dataLocations);
}
}
}

五、应用级容错：ApplicationMaster恢复

5.1 AM失效的影响

ApplicationMaster是整个作业的"大脑"，如果AM失效，整个作业将无法继续：

• Map任务完成但无法通知Reduce
• Reduce任务无法启动
• 作业进度丢失

5.2 AM恢复机制

Task

AM2

NodeManager

ZooKeeper

ResourceManager

ApplicationMaster

Task

AM2

NodeManager

ZooKeeper

ResourceManager

ApplicationMaster

#mermaid-svg-6b2D57RwUeEXIiWU{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-6b2D57RwUeEXIiWU .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-6b2D57RwUeEXIiWU .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-6b2D57RwUeEXIiWU .error-icon{fill:#552222;}#mermaid-svg-6b2D57RwUeEXIiWU .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-6b2D57RwUeEXIiWU .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-6b2D57RwUeEXIiWU .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-6b2D57RwUeEXIiWU .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-6b2D57RwUeEXIiWU .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-6b2D57RwUeEXIiWU .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-6b2D57RwUeEXIiWU .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-6b2D57RwUeEXIiWU .marker{fill:#333333;stroke:#333333;}#mermaid-svg-6b2D57RwUeEXIiWU .marker.cross{stroke:#333333;}#mermaid-svg-6b2D57RwUeEXIiWU svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-6b2D57RwUeEXIiWU p{margin:0;}#mermaid-svg-6b2D57RwUeEXIiWU .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-6b2D57RwUeEXIiWU text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-6b2D57RwUeEXIiWU .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-6b2D57RwUeEXIiWU .innerArc{stroke-width:1.5;stroke-dasharray:none;}#mermaid-svg-6b2D57RwUeEXIiWU .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-6b2D57RwUeEXIiWU .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-6b2D57RwUeEXIiWU #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-6b2D57RwUeEXIiWU .sequenceNumber{fill:white;}#mermaid-svg-6b2D57RwUeEXIiWU #sequencenumber{fill:#333;}#mermaid-svg-6b2D57RwUeEXIiWU #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-6b2D57RwUeEXIiWU .messageText{fill:#333;stroke:none;}#mermaid-svg-6b2D57RwUeEXIiWU .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-6b2D57RwUeEXIiWU .labelText,#mermaid-svg-6b2D57RwUeEXIiWU .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-6b2D57RwUeEXIiWU .loopText,#mermaid-svg-6b2D57RwUeEXIiWU .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-6b2D57RwUeEXIiWU .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-6b2D57RwUeEXIiWU .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-6b2D57RwUeEXIiWU .noteText,#mermaid-svg-6b2D57RwUeEXIiWU .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-6b2D57RwUeEXIiWU .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-6b2D57RwUeEXIiWU .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-6b2D57RwUeEXIiWU .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-6b2D57RwUeEXIiWU .actorPopupMenu{position:absolute;}#mermaid-svg-6b2D57RwUeEXIiWU .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-6b2D57RwUeEXIiWU .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-6b2D57RwUeEXIiWU .actor-man circle,#mermaid-svg-6b2D57RwUeEXIiWU line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-6b2D57RwUeEXIiWU :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

AM崩溃

检测到AM心跳丢失

记录AM状态到ZK

在新节点启动新AM

启动新ApplicationMaster

从ZK读取作业状态

重建作业状态

重新注册

获取运行中任务状态

恢复作业执行

5.3 AM恢复配置

<!– yarn-site.xml AM恢复配置 –>
<property>
<name>yarn.resourcemanager.am.max-attempts</name>
<value>2</value>
<description>ApplicationMaster最大尝试次数</description>
</property>

<!– 启用RM恢复 –>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>

<!– 使用ZooKeeper存储状态 –>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>

<property>
<name>yarn.resourcemanager.zk-state-store.address</name>
<value>zk1:2181,zk2:2181,zk3:2181</value>
</property>

六、集群级容错：HA架构

6.1 NameNode HA

NameNode是HDFS的单点故障，HA架构通过两个NameNode解决：

#mermaid-svg-VDRsoI9JeqKuQDN4{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-VDRsoI9JeqKuQDN4 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-VDRsoI9JeqKuQDN4 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-VDRsoI9JeqKuQDN4 .error-icon{fill:#552222;}#mermaid-svg-VDRsoI9JeqKuQDN4 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-VDRsoI9JeqKuQDN4 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-VDRsoI9JeqKuQDN4 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-VDRsoI9JeqKuQDN4 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-VDRsoI9JeqKuQDN4 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-VDRsoI9JeqKuQDN4 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-VDRsoI9JeqKuQDN4 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-VDRsoI9JeqKuQDN4 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-VDRsoI9JeqKuQDN4 .marker.cross{stroke:#333333;}#mermaid-svg-VDRsoI9JeqKuQDN4 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-VDRsoI9JeqKuQDN4 p{margin:0;}#mermaid-svg-VDRsoI9JeqKuQDN4 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-VDRsoI9JeqKuQDN4 .cluster-label text{fill:#333;}#mermaid-svg-VDRsoI9JeqKuQDN4 .cluster-label span{color:#333;}#mermaid-svg-VDRsoI9JeqKuQDN4 .cluster-label span p{background-color:transparent;}#mermaid-svg-VDRsoI9JeqKuQDN4 .label text,#mermaid-svg-VDRsoI9JeqKuQDN4 span{fill:#333;color:#333;}#mermaid-svg-VDRsoI9JeqKuQDN4 .node rect,#mermaid-svg-VDRsoI9JeqKuQDN4 .node circle,#mermaid-svg-VDRsoI9JeqKuQDN4 .node ellipse,#mermaid-svg-VDRsoI9JeqKuQDN4 .node polygon,#mermaid-svg-VDRsoI9JeqKuQDN4 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-VDRsoI9JeqKuQDN4 .rough-node .label text,#mermaid-svg-VDRsoI9JeqKuQDN4 .node .label text,#mermaid-svg-VDRsoI9JeqKuQDN4 .image-shape .label,#mermaid-svg-VDRsoI9JeqKuQDN4 .icon-shape .label{text-anchor:middle;}#mermaid-svg-VDRsoI9JeqKuQDN4 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-VDRsoI9JeqKuQDN4 .rough-node .label,#mermaid-svg-VDRsoI9JeqKuQDN4 .node .label,#mermaid-svg-VDRsoI9JeqKuQDN4 .image-shape .label,#mermaid-svg-VDRsoI9JeqKuQDN4 .icon-shape .label{text-align:center;}#mermaid-svg-VDRsoI9JeqKuQDN4 .node.clickable{cursor:pointer;}#mermaid-svg-VDRsoI9JeqKuQDN4 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-VDRsoI9JeqKuQDN4 .arrowheadPath{fill:#333333;}#mermaid-svg-VDRsoI9JeqKuQDN4 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-VDRsoI9JeqKuQDN4 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-VDRsoI9JeqKuQDN4 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-VDRsoI9JeqKuQDN4 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-VDRsoI9JeqKuQDN4 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-VDRsoI9JeqKuQDN4 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-VDRsoI9JeqKuQDN4 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-VDRsoI9JeqKuQDN4 .cluster text{fill:#333;}#mermaid-svg-VDRsoI9JeqKuQDN4 .cluster span{color:#333;}#mermaid-svg-VDRsoI9JeqKuQDN4 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-VDRsoI9JeqKuQDN4 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-VDRsoI9JeqKuQDN4 rect.text{fill:none;stroke-width:0;}#mermaid-svg-VDRsoI9JeqKuQDN4 .icon-shape,#mermaid-svg-VDRsoI9JeqKuQDN4 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-VDRsoI9JeqKuQDN4 .icon-shape p,#mermaid-svg-VDRsoI9JeqKuQDN4 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-VDRsoI9JeqKuQDN4 .icon-shape rect,#mermaid-svg-VDRsoI9JeqKuQDN4 .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-VDRsoI9JeqKuQDN4 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-VDRsoI9JeqKuQDN4 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-VDRsoI9JeqKuQDN4 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

NameNode HA架构

写入EditLog

写入EditLog

写入EditLog

读取EditLog

读取EditLog

读取EditLog

监控

监控

NameNode Active

JournalNode 1

JournalNode 2

JournalNode 3

NameNode Standby

ZooKeeper集群

6.2 ResourceManager HA

类似地，ResourceManager也支持HA：

<!– yarn-site.xml RM HA配置 –>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>

<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yarn-cluster</value>
</property>

<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>

<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>rm1-host</value>
</property>

<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>rm2-host</value>
</property>

<property>
<name>yarn.resourcemanager.zk-address</name>
<value>zk1:2181,zk2:2181,zk3:2181</value>
</property>

七、容错机制的监控与调优

7.1 关键监控指标

# 查看任务失败情况
yarn application -status <application-id> | grep -A 10 "Task"

# 通过Job History Server查看失败任务
# http://job-history-server:19888

# 查看AM重试次数
hadoop job -history <job-output-dir>

# 实时监控
watch -n 5 'yarn node -list | grep -i "status"'

7.2 常见问题排查

7.3 调优建议

八、总结：容错机制的最佳实践

8.1 容错体系核心要点

#mermaid-svg-PsA8sJ3JXsSd4Z93{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .error-icon{fill:#552222;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .marker.cross{stroke:#333333;}#mermaid-svg-PsA8sJ3JXsSd4Z93 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-PsA8sJ3JXsSd4Z93 p{margin:0;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge{stroke-width:3;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section–1 rect,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section–1 path,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section–1 circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section–1 polygon,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section–1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section–1 text{fill:#ffffff;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .node-icon–1{font-size:40px;color:#ffffff;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-edge–1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-depth–1{stroke-width:17;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section–1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:lightgray;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:#efefef;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-0 rect,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-0 path,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-0 circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-0 polygon,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-0 text{fill:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .node-icon-0{font-size:40px;color:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-depth-0{stroke-width:14;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:lightgray;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:#efefef;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-1 rect,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-1 path,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-1 circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-1 polygon,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-1 text{fill:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .node-icon-1{font-size:40px;color:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-depth-1{stroke-width:11;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:lightgray;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:#efefef;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-2 rect,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-2 path,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-2 circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-2 polygon,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-2 text{fill:#ffffff;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-depth-2{stroke-width:8;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:lightgray;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:#efefef;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-3 rect,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-3 path,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-3 circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-3 polygon,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-3 text{fill:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .node-icon-3{font-size:40px;color:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-depth-3{stroke-width:5;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:lightgray;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:#efefef;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-4 rect,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-4 path,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-4 circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-4 polygon,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-4 text{fill:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .node-icon-4{font-size:40px;color:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-depth-4{stroke-width:2;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:lightgray;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:#efefef;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-5 rect,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-5 path,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-5 circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-5 polygon,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-5 text{fill:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .node-icon-5{font-size:40px;color:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-depth-5{stroke-width:-1;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:lightgray;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:#efefef;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-6 rect,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-6 path,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-6 circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-6 polygon,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-6 text{fill:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .node-icon-6{font-size:40px;color:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-depth-6{stroke-width:-4;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:lightgray;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:#efefef;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-7 rect,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-7 path,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-7 circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-7 polygon,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-7 text{fill:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .node-icon-7{font-size:40px;color:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-depth-7{stroke-width:-7;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:lightgray;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:#efefef;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-8 rect,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-8 path,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-8 circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-8 polygon,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-8 text{fill:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .node-icon-8{font-size:40px;color:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-depth-8{stroke-width:-10;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:lightgray;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:#efefef;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-9 rect,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-9 path,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-9 circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-9 polygon,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-9 text{fill:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .node-icon-9{font-size:40px;color:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-depth-9{stroke-width:-13;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:lightgray;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:#efefef;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-10 rect,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-10 path,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-10 circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-10 polygon,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-10 text{fill:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .node-icon-10{font-size:40px;color:black;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge-depth-10{stroke-width:-16;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:lightgray;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .disabled text{fill:#efefef;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-root rect,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-root path,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-root circle,#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-root text{fill:#ffffff;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-root span{color:#ffffff;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .section-2 span{color:#ffffff;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .edge{fill:none;}#mermaid-svg-PsA8sJ3JXsSd4Z93 .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-PsA8sJ3JXsSd4Z93 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Hadoop容错

检测层

心跳监控

超时检测

健康检查

恢复层

任务重试

节点迁移

AM重启

HA切换

预防层

数据冗余

推测执行

资源隔离

8.2 黄金法则

“容错不是万能的，没有容错是万万不能的。合理的容错配置，能在稳定性和效率之间找到最佳平衡。”

8.3 最终建议

对于大多数生产环境，建议采用以下基准配置：

互动问题：你在实际工作中遇到过哪些棘手的任务失败问题？是重试机制配置不当导致的效率问题，还是节点故障引发的连锁反应？欢迎在评论区分享你的经验和解决方案！