Hadoop MapReduce机制详解

Hadoop MapReduce机制详解

概述

Hadoop MapReduce是一个分布式计算框架，用于处理大规模数据集。它基于Google的MapReduce论文实现，将复杂的分布式计算抽象为两个主要阶段：Map阶段和Reduce阶段。

核心架构

1. 主要组件

JobTracker

• 职责：作业调度与监控
• 功能：
  • 接收客户端提交的作业
  • 将作业分解为多个任务
  • 调度任务到可用的TaskTracker
  • 监控任务执行状态
  • 重新执行失败的任务

TaskTracker

• 职责：任务执行
• 功能：
  • 接收JobTracker分配的任务
  • 在本地执行Map或Reduce任务
  • 定期向JobTracker发送心跳信号
  • 报告任务执行进度和状态

JobClient

• 职责：作业提交接口
• 功能：
  • 配置作业参数
  • 上传作业所需的JAR文件和配置
  • 提交作业到JobTracker
  • 监控作业执行状态

2. 数据存储层 – HDFS

MapReduce通常与HDFS（Hadoop Distributed File System）配合使用：

#mermaid-svg-8RKNY1taLhFi1kei {font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-8RKNY1taLhFi1kei .error-icon{fill:#552222;}#mermaid-svg-8RKNY1taLhFi1kei .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-8RKNY1taLhFi1kei .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-8RKNY1taLhFi1kei .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-8RKNY1taLhFi1kei .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-8RKNY1taLhFi1kei .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-8RKNY1taLhFi1kei .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-8RKNY1taLhFi1kei .marker{fill:#333333;stroke:#333333;}#mermaid-svg-8RKNY1taLhFi1kei .marker.cross{stroke:#333333;}#mermaid-svg-8RKNY1taLhFi1kei svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-8RKNY1taLhFi1kei .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-8RKNY1taLhFi1kei .cluster-label text{fill:#333;}#mermaid-svg-8RKNY1taLhFi1kei .cluster-label span{color:#333;}#mermaid-svg-8RKNY1taLhFi1kei .label text,#mermaid-svg-8RKNY1taLhFi1kei span{fill:#333;color:#333;}#mermaid-svg-8RKNY1taLhFi1kei .node rect,#mermaid-svg-8RKNY1taLhFi1kei .node circle,#mermaid-svg-8RKNY1taLhFi1kei .node ellipse,#mermaid-svg-8RKNY1taLhFi1kei .node polygon,#mermaid-svg-8RKNY1taLhFi1kei .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-8RKNY1taLhFi1kei .node .label{text-align:center;}#mermaid-svg-8RKNY1taLhFi1kei .node.clickable{cursor:pointer;}#mermaid-svg-8RKNY1taLhFi1kei .arrowheadPath{fill:#333333;}#mermaid-svg-8RKNY1taLhFi1kei .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-8RKNY1taLhFi1kei .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-8RKNY1taLhFi1kei .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-svg-8RKNY1taLhFi1kei .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-svg-8RKNY1taLhFi1kei .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-8RKNY1taLhFi1kei .cluster text{fill:#333;}#mermaid-svg-8RKNY1taLhFi1kei .cluster span{color:#333;}#mermaid-svg-8RKNY1taLhFi1kei div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-8RKNY1taLhFi1kei :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}提交作业分配任务分配任务分配任务读取数据读取数据读取数据本地计算本地计算本地计算客户端JobTrackerTaskTracker1TaskTracker2TaskTracker3HDFS DataNode1HDFS DataNode2HDFS DataNode3

MapReduce执行流程

1. 作业提交阶段

#mermaid-svg-B3TSgE50Q6qB4uQO {font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-B3TSgE50Q6qB4uQO .error-icon{fill:#552222;}#mermaid-svg-B3TSgE50Q6qB4uQO .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-B3TSgE50Q6qB4uQO .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-B3TSgE50Q6qB4uQO .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-B3TSgE50Q6qB4uQO .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-B3TSgE50Q6qB4uQO .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-B3TSgE50Q6qB4uQO .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-B3TSgE50Q6qB4uQO .marker{fill:#333333;stroke:#333333;}#mermaid-svg-B3TSgE50Q6qB4uQO .marker.cross{stroke:#333333;}#mermaid-svg-B3TSgE50Q6qB4uQO svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-B3TSgE50Q6qB4uQO .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-B3TSgE50Q6qB4uQO text.actor>tspan{fill:black;stroke:none;}#mermaid-svg-B3TSgE50Q6qB4uQO .actor-line{stroke:grey;}#mermaid-svg-B3TSgE50Q6qB4uQO .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-svg-B3TSgE50Q6qB4uQO .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-svg-B3TSgE50Q6qB4uQO #arrowhead path{fill:#333;stroke:#333;}#mermaid-svg-B3TSgE50Q6qB4uQO .sequenceNumber{fill:white;}#mermaid-svg-B3TSgE50Q6qB4uQO #sequencenumber{fill:#333;}#mermaid-svg-B3TSgE50Q6qB4uQO #crosshead path{fill:#333;stroke:#333;}#mermaid-svg-B3TSgE50Q6qB4uQO .messageText{fill:#333;stroke:#333;}#mermaid-svg-B3TSgE50Q6qB4uQO .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-B3TSgE50Q6qB4uQO .labelText,#mermaid-svg-B3TSgE50Q6qB4uQO .labelText>tspan{fill:black;stroke:none;}#mermaid-svg-B3TSgE50Q6qB4uQO .loopText,#mermaid-svg-B3TSgE50Q6qB4uQO .loopText>tspan{fill:black;stroke:none;}#mermaid-svg-B3TSgE50Q6qB4uQO .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#mermaid-svg-B3TSgE50Q6qB4uQO .note{stroke:#aaaa33;fill:#fff5ad;}#mermaid-svg-B3TSgE50Q6qB4uQO .noteText,#mermaid-svg-B3TSgE50Q6qB4uQO .noteText>tspan{fill:black;stroke:none;}#mermaid-svg-B3TSgE50Q6qB4uQO .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-svg-B3TSgE50Q6qB4uQO .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-svg-B3TSgE50Q6qB4uQO .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-svg-B3TSgE50Q6qB4uQO .actorPopupMenu{position:absolute;}#mermaid-svg-B3TSgE50Q6qB4uQO .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-svg-B3TSgE50Q6qB4uQO .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#mermaid-svg-B3TSgE50Q6qB4uQO .actor-man circle,#mermaid-svg-B3TSgE50Q6qB4uQO line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#mermaid-svg-B3TSgE50Q6qB4uQO :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}ClientJobTrackerHDFSTaskTracker上传输入数据提交作业请求获取输入数据信息计算输入分片返回作业ID上传作业JAR和配置ClientJobTrackerHDFSTaskTracker

2. Map阶段详解

输入分片（Input Split）

• 将输入数据逻辑分割成多个分片
• 每个分片对应一个Map任务
• 分片大小通常等于HDFS块大小（默认128MB）

Map函数执行

public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);

while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
context.write(word, one);
}
}
}

Map阶段输出处理

3. Shuffle和Sort阶段

Shuffle过程

#mermaid-svg-208LFZIGrEcM02jf {font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-208LFZIGrEcM02jf .error-icon{fill:#552222;}#mermaid-svg-208LFZIGrEcM02jf .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-208LFZIGrEcM02jf .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-208LFZIGrEcM02jf .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-208LFZIGrEcM02jf .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-208LFZIGrEcM02jf .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-208LFZIGrEcM02jf .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-208LFZIGrEcM02jf .marker{fill:#333333;stroke:#333333;}#mermaid-svg-208LFZIGrEcM02jf .marker.cross{stroke:#333333;}#mermaid-svg-208LFZIGrEcM02jf svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-208LFZIGrEcM02jf .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-208LFZIGrEcM02jf .cluster-label text{fill:#333;}#mermaid-svg-208LFZIGrEcM02jf .cluster-label span{color:#333;}#mermaid-svg-208LFZIGrEcM02jf .label text,#mermaid-svg-208LFZIGrEcM02jf span{fill:#333;color:#333;}#mermaid-svg-208LFZIGrEcM02jf .node rect,#mermaid-svg-208LFZIGrEcM02jf .node circle,#mermaid-svg-208LFZIGrEcM02jf .node ellipse,#mermaid-svg-208LFZIGrEcM02jf .node polygon,#mermaid-svg-208LFZIGrEcM02jf .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-208LFZIGrEcM02jf .node .label{text-align:center;}#mermaid-svg-208LFZIGrEcM02jf .node.clickable{cursor:pointer;}#mermaid-svg-208LFZIGrEcM02jf .arrowheadPath{fill:#333333;}#mermaid-svg-208LFZIGrEcM02jf .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-208LFZIGrEcM02jf .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-208LFZIGrEcM02jf .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-svg-208LFZIGrEcM02jf .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-svg-208LFZIGrEcM02jf .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-208LFZIGrEcM02jf .cluster text{fill:#333;}#mermaid-svg-208LFZIGrEcM02jf .cluster span{color:#333;}#mermaid-svg-208LFZIGrEcM02jf div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-208LFZIGrEcM02jf :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}分区1分区2分区1分区2分区1分区2排序合并排序合并Map Task 1Reducer 1Reducer 2Map Task 2Map Task 3Reduce Task 1Reduce Task 2

详细步骤

4. Reduce阶段

Reduce函数执行

public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}

Reduce阶段特点

• 接收分组后的键值对
• 对每个key的所有values进行处理
• 输出最终结果到HDFS

容错机制

1. 任务失败处理

Map任务失败

• JobTracker检测到TaskTracker心跳超时
• 将该TaskTracker标记为失败
• 重新调度该TaskTracker上的所有任务
• 在其他节点重新执行

Reduce任务失败

• 类似Map任务失败处理
• 需要重新获取Map输出数据
• 重新执行Reduce计算

2. 推测执行（Speculative Execution）

#mermaid-svg-hw9FH4t9HG1RfSQu {font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-hw9FH4t9HG1RfSQu .error-icon{fill:#552222;}#mermaid-svg-hw9FH4t9HG1RfSQu .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-hw9FH4t9HG1RfSQu .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-hw9FH4t9HG1RfSQu .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-hw9FH4t9HG1RfSQu .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-hw9FH4t9HG1RfSQu .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-hw9FH4t9HG1RfSQu .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-hw9FH4t9HG1RfSQu .marker{fill:#333333;stroke:#333333;}#mermaid-svg-hw9FH4t9HG1RfSQu .marker.cross{stroke:#333333;}#mermaid-svg-hw9FH4t9HG1RfSQu svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-hw9FH4t9HG1RfSQu .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-hw9FH4t9HG1RfSQu .cluster-label text{fill:#333;}#mermaid-svg-hw9FH4t9HG1RfSQu .cluster-label span{color:#333;}#mermaid-svg-hw9FH4t9HG1RfSQu .label text,#mermaid-svg-hw9FH4t9HG1RfSQu span{fill:#333;color:#333;}#mermaid-svg-hw9FH4t9HG1RfSQu .node rect,#mermaid-svg-hw9FH4t9HG1RfSQu .node circle,#mermaid-svg-hw9FH4t9HG1RfSQu .node ellipse,#mermaid-svg-hw9FH4t9HG1RfSQu .node polygon,#mermaid-svg-hw9FH4t9HG1RfSQu .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-hw9FH4t9HG1RfSQu .node .label{text-align:center;}#mermaid-svg-hw9FH4t9HG1RfSQu .node.clickable{cursor:pointer;}#mermaid-svg-hw9FH4t9HG1RfSQu .arrowheadPath{fill:#333333;}#mermaid-svg-hw9FH4t9HG1RfSQu .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-hw9FH4t9HG1RfSQu .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-hw9FH4t9HG1RfSQu .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-svg-hw9FH4t9HG1RfSQu .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-svg-hw9FH4t9HG1RfSQu .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-hw9FH4t9HG1RfSQu .cluster text{fill:#333;}#mermaid-svg-hw9FH4t9HG1RfSQu .cluster span{color:#333;}#mermaid-svg-hw9FH4t9HG1RfSQu div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-hw9FH4t9HG1RfSQu :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}原任务备份任务任务执行缓慢启动备份任务备份任务执行哪个任务先完成?终止备份任务终止原任务

• 检测执行缓慢的任务
• 在空闲节点启动备份任务
• 采用先完成的结果
• 提高整体作业完成时间

3. 数据本地化优化

本地化级别

调度策略

• 优先选择节点本地化
• 其次选择机架本地化
• 最后考虑全局本地化
• 减少网络传输开销

性能优化策略

1. Combiner使用

job.setCombinerClass(WordCountReducer.class);

• 减少Map输出数据量
• 降低网络传输负载
• 仅适用于可交换和可结合的操作

2. 压缩配置

<!– 启用Map输出压缩 –>
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>

<property>
<name>mapreduce.map.output.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

3. 内存调优

<!– Map任务内存 –>
<property>
<name>mapreduce.map.memory.mb</name>
<value>2048</value>
</property>

<!– Reduce任务内存 –>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>4096</value>
</property>

实际应用案例

1. 日志分析

public class LogAnalyzer {
// Map函数：提取IP地址和访问时间
public static class LogMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
private static final IntWritable one = new IntWritable(1);
private Text ipAddress = new Text();

public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
String ip = extractIP(line); // 自定义IP提取方法
if (ip != null) {
ipAddress.set(ip);
context.write(ipAddress, one);
}
}
}

// Reduce函数：统计每个IP的访问次数
public static class LogReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
}

2. 数据去重

public class DataDeduplication {
public static class DedupMapper extends Mapper<LongWritable, Text, Text, NullWritable> {
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
// 直接输出整行作为key
context.write(value, NullWritable.get());
}
}

public static class DedupReducer extends Reducer<Text, NullWritable, Text, NullWritable> {
public void reduce(Text key, Iterable<NullWritable> values, Context context)
throws IOException, InterruptedException {
// 相同key只输出一次
context.write(key, NullWritable.get());
}
}
}

优缺点分析

优点

缺点

与其他框架对比

总结

Hadoop MapReduce作为大数据处理的开山之作，为分布式计算提供了简单而强大的编程模型。虽然在某些场景下已被更现代的框架如Spark所取代，但其在数据本地化、容错机制和大规模批处理方面的设计理念仍然影响着现代大数据处理系统。理解MapReduce的工作原理对于深入学习大数据技术栈具有重要意义。