Hadoop MapReduce编程模型深度解析：从分片到输出的完整流程

Hadoop MapReduce编程模型深度解析：从分片到输出的完整流程

• • 引言
  • 一、MapReduce整体流程概览
  • • 1.1 你的描述验证
  • 二、Map阶段详解
  • • 2.1 输入分片与读取
    • 2.2 Mapper实现
    • 2.3 Hadoop内置数据类型
  • 三、分区（Partition）阶段
  • • 3.1 默认HashPartitioner
    • 3.2 自定义分区器
    • 3.3 分区示意图
  • 四、排序（Sort）阶段
  • • 4.1 默认排序规则
    • 4.2 自定义排序
  • 五、分组（Group）阶段
  • • 5.1 默认分组规则
    • 5.2 自定义分组
  • 六、Combiner归约阶段
  • • 6.1 Combiner作用
    • 6.2 Combiner效果对比
  • 七、Reduce阶段
  • • 7.1 Reducer实现
    • 7.2 Shuffle过程详解
  • 八、完整WordCount示例
  • • 8.1 主类实现
  • 九、执行流程总结
  • 十、总结

引言

MapReduce是Hadoop的核心计算模型，它通过分而治之的思想，将大规模数据处理任务分解为可并行执行的Map和Reduce阶段。理解MapReduce的工作流程，是掌握Hadoop编程的基础。本文将基于你的描述，详细解析MapReduce的完整执行流程。

一、MapReduce整体流程概览

1.1 你的描述验证

你描述的流程非常准确，完整涵盖了MapReduce的各个阶段：

#mermaid-svg-Pv5MfymUpxk28ZVi{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Pv5MfymUpxk28ZVi .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Pv5MfymUpxk28ZVi .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Pv5MfymUpxk28ZVi .error-icon{fill:#552222;}#mermaid-svg-Pv5MfymUpxk28ZVi .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Pv5MfymUpxk28ZVi .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Pv5MfymUpxk28ZVi .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Pv5MfymUpxk28ZVi .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Pv5MfymUpxk28ZVi .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Pv5MfymUpxk28ZVi .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Pv5MfymUpxk28ZVi .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Pv5MfymUpxk28ZVi .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Pv5MfymUpxk28ZVi .marker.cross{stroke:#333333;}#mermaid-svg-Pv5MfymUpxk28ZVi svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Pv5MfymUpxk28ZVi p{margin:0;}#mermaid-svg-Pv5MfymUpxk28ZVi .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-Pv5MfymUpxk28ZVi .cluster-label text{fill:#333;}#mermaid-svg-Pv5MfymUpxk28ZVi .cluster-label span{color:#333;}#mermaid-svg-Pv5MfymUpxk28ZVi .cluster-label span p{background-color:transparent;}#mermaid-svg-Pv5MfymUpxk28ZVi .label text,#mermaid-svg-Pv5MfymUpxk28ZVi span{fill:#333;color:#333;}#mermaid-svg-Pv5MfymUpxk28ZVi .node rect,#mermaid-svg-Pv5MfymUpxk28ZVi .node circle,#mermaid-svg-Pv5MfymUpxk28ZVi .node ellipse,#mermaid-svg-Pv5MfymUpxk28ZVi .node polygon,#mermaid-svg-Pv5MfymUpxk28ZVi .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Pv5MfymUpxk28ZVi .rough-node .label text,#mermaid-svg-Pv5MfymUpxk28ZVi .node .label text,#mermaid-svg-Pv5MfymUpxk28ZVi .image-shape .label,#mermaid-svg-Pv5MfymUpxk28ZVi .icon-shape .label{text-anchor:middle;}#mermaid-svg-Pv5MfymUpxk28ZVi .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Pv5MfymUpxk28ZVi .rough-node .label,#mermaid-svg-Pv5MfymUpxk28ZVi .node .label,#mermaid-svg-Pv5MfymUpxk28ZVi .image-shape .label,#mermaid-svg-Pv5MfymUpxk28ZVi .icon-shape .label{text-align:center;}#mermaid-svg-Pv5MfymUpxk28ZVi .node.clickable{cursor:pointer;}#mermaid-svg-Pv5MfymUpxk28ZVi .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Pv5MfymUpxk28ZVi .arrowheadPath{fill:#333333;}#mermaid-svg-Pv5MfymUpxk28ZVi .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Pv5MfymUpxk28ZVi .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Pv5MfymUpxk28ZVi .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Pv5MfymUpxk28ZVi .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Pv5MfymUpxk28ZVi .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Pv5MfymUpxk28ZVi .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Pv5MfymUpxk28ZVi .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Pv5MfymUpxk28ZVi .cluster text{fill:#333;}#mermaid-svg-Pv5MfymUpxk28ZVi .cluster span{color:#333;}#mermaid-svg-Pv5MfymUpxk28ZVi div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Pv5MfymUpxk28ZVi .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Pv5MfymUpxk28ZVi rect.text{fill:none;stroke-width:0;}#mermaid-svg-Pv5MfymUpxk28ZVi .icon-shape,#mermaid-svg-Pv5MfymUpxk28ZVi .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Pv5MfymUpxk28ZVi .icon-shape p,#mermaid-svg-Pv5MfymUpxk28ZVi .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Pv5MfymUpxk28ZVi .icon-shape rect,#mermaid-svg-Pv5MfymUpxk28ZVi .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Pv5MfymUpxk28ZVi .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Pv5MfymUpxk28ZVi .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Pv5MfymUpxk28ZVi :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

读取数据

键值对

分区数据

排序后

分组后

本地聚合

网络传输

最终结果

本地文件系统

Map阶段

Partition分区

Sort排序

Group分组

Combiner归约可选

Shuffle传输

Reduce阶段

输出保存

二、Map阶段详解

2.1 输入分片与读取

public class MapStage {

// 1. 输入分片（InputSplit）
// MapReduce将输入文件切分成多个分片，每个分片由一个Map处理

// 2. RecordReader读取数据
// 将分片中的数据解析成键值对

// 默认的TextInputFormat
// 键: 行偏移量 (LongWritable)
// 值: 行内容 (Text)

// 输入文件内容示例
/*
hello world
hadoop mapreduce
*/

// 转换为键值对
// <0, "hello world">
// <12, "hadoop mapreduce">
}

2.2 Mapper实现

public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

@Override
protected void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {

// 获取一行数据
String line = value.toString();

// 按空格分词
String[] words = line.split(" ");

// 输出每个单词，计数为1
for (String w : words) {
word.set(w);
context.write(word, one); // 输出 <word, 1>
}
}
}

2.3 Hadoop内置数据类型

三、分区（Partition）阶段

3.1 默认HashPartitioner

public class HashPartitioner<K, V> extends Partitioner<K, V> {

// 默认分区规则：根据key的hashCode取模
public int getPartition(K key, V value, int numReduceTasks) {
return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks;
}

// 例如：有3个Reducer，key="hello"
// hashcode = 99162322
// 99162322 % 3 = 1 → 分配到第1个Reducer
}

3.2 自定义分区器

public class CustomPartitioner extends Partitioner<Text, IntWritable> {

@Override
public int getPartition(Text key, IntWritable value, int numPartitions) {
String word = key.toString();

// 根据单词首字母分区
char firstChar = word.charAt(0);

if (firstChar >= 'a' && firstChar <= 'z') {
return (firstChar – 'a') % numPartitions;
} else {
return 0; // 其他字符统一分区
}
}
}

// 在Job中设置自定义分区器
job.setPartitionerClass(CustomPartitioner.class);

3.3 分区示意图

#mermaid-svg-hBjvyfahWQE17THz{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-hBjvyfahWQE17THz .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-hBjvyfahWQE17THz .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-hBjvyfahWQE17THz .error-icon{fill:#552222;}#mermaid-svg-hBjvyfahWQE17THz .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-hBjvyfahWQE17THz .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-hBjvyfahWQE17THz .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-hBjvyfahWQE17THz .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-hBjvyfahWQE17THz .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-hBjvyfahWQE17THz .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-hBjvyfahWQE17THz .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-hBjvyfahWQE17THz .marker{fill:#333333;stroke:#333333;}#mermaid-svg-hBjvyfahWQE17THz .marker.cross{stroke:#333333;}#mermaid-svg-hBjvyfahWQE17THz svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-hBjvyfahWQE17THz p{margin:0;}#mermaid-svg-hBjvyfahWQE17THz .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-hBjvyfahWQE17THz .cluster-label text{fill:#333;}#mermaid-svg-hBjvyfahWQE17THz .cluster-label span{color:#333;}#mermaid-svg-hBjvyfahWQE17THz .cluster-label span p{background-color:transparent;}#mermaid-svg-hBjvyfahWQE17THz .label text,#mermaid-svg-hBjvyfahWQE17THz span{fill:#333;color:#333;}#mermaid-svg-hBjvyfahWQE17THz .node rect,#mermaid-svg-hBjvyfahWQE17THz .node circle,#mermaid-svg-hBjvyfahWQE17THz .node ellipse,#mermaid-svg-hBjvyfahWQE17THz .node polygon,#mermaid-svg-hBjvyfahWQE17THz .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-hBjvyfahWQE17THz .rough-node .label text,#mermaid-svg-hBjvyfahWQE17THz .node .label text,#mermaid-svg-hBjvyfahWQE17THz .image-shape .label,#mermaid-svg-hBjvyfahWQE17THz .icon-shape .label{text-anchor:middle;}#mermaid-svg-hBjvyfahWQE17THz .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-hBjvyfahWQE17THz .rough-node .label,#mermaid-svg-hBjvyfahWQE17THz .node .label,#mermaid-svg-hBjvyfahWQE17THz .image-shape .label,#mermaid-svg-hBjvyfahWQE17THz .icon-shape .label{text-align:center;}#mermaid-svg-hBjvyfahWQE17THz .node.clickable{cursor:pointer;}#mermaid-svg-hBjvyfahWQE17THz .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-hBjvyfahWQE17THz .arrowheadPath{fill:#333333;}#mermaid-svg-hBjvyfahWQE17THz .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-hBjvyfahWQE17THz .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-hBjvyfahWQE17THz .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hBjvyfahWQE17THz .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-hBjvyfahWQE17THz .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hBjvyfahWQE17THz .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-hBjvyfahWQE17THz .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-hBjvyfahWQE17THz .cluster text{fill:#333;}#mermaid-svg-hBjvyfahWQE17THz .cluster span{color:#333;}#mermaid-svg-hBjvyfahWQE17THz div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-hBjvyfahWQE17THz .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-hBjvyfahWQE17THz rect.text{fill:none;stroke-width:0;}#mermaid-svg-hBjvyfahWQE17THz .icon-shape,#mermaid-svg-hBjvyfahWQE17THz .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hBjvyfahWQE17THz .icon-shape p,#mermaid-svg-hBjvyfahWQE17THz .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-hBjvyfahWQE17THz .icon-shape rect,#mermaid-svg-hBjvyfahWQE17THz .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hBjvyfahWQE17THz .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-hBjvyfahWQE17THz .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-hBjvyfahWQE17THz :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Map输出

hashCode%3=0

hashCode%3=1

hashCode%3=2

分区器

Reducer 1apple, cat

Reducer 2banana

Reducer 3dog

四、排序（Sort）阶段

4.1 默认排序规则

// MapReduce默认按key进行字典序排序
// 输入: <apple,1>, <banana,1>, <cat,1>
// 排序后: <apple,1>, <banana,1>, <cat,1>

4.2 自定义排序

// 自定义数据类型实现WritableComparable
public class PersonWritable implements WritableComparable<PersonWritable> {

private String name;
private int age;

@Override
public int compareTo(PersonWritable o) {
// 先按年龄排序，再按姓名排序
if (this.age != o.age) {
return this.age – o.age; // 年龄升序
} else {
return this.name.compareTo(o.name); // 姓名升序
}
}
}

// 或者继承WritableComparator
public class PersonComparator extends WritableComparator {

protected PersonComparator() {
super(PersonWritable.class, true);
}

@Override
public int compare(WritableComparable a, WritableComparable b) {
PersonWritable p1 = (PersonWritable) a;
PersonWritable p2 = (PersonWritable) b;
return p1.compareTo(p2);
}
}

五、分组（Group）阶段

5.1 默认分组规则

// 默认情况下，分组与排序规则一致
// 相同key的值会被分到同一组

// 输入: <apple,1>, <apple,1>, <banana,1>
// 分组后:
// 组1: apple -> [1,1]
// 组2: banana -> [1]

5.2 自定义分组

// 自定义分组比较器
public class CustomGroupComparator extends WritableComparator {

protected CustomGroupComparator() {
super(Text.class, true);
}

@Override
public int compare(WritableComparable a, WritableComparable b) {
Text t1 = (Text) a;
Text t2 = (Text) b;

// 只按首字母分组
String s1 = t1.toString();
String s2 = t2.toString();

char c1 = s1.charAt(0);
char c2 = s2.charAt(0);

return Character.compare(c1, c2);
}
}

// 在Job中设置
job.setGroupingComparatorClass(CustomGroupComparator.class);

// 效果：apple和apricot会分到同一组，即使key不同

六、Combiner归约阶段

6.1 Combiner作用

public class WordCountCombiner extends Reducer<Text, IntWritable, Text, IntWritable> {

private IntWritable result = new IntWritable();

@Override
protected void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {

int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}

// 在Job中设置
job.setCombinerClass(WordCountCombiner.class);

6.2 Combiner效果对比

#mermaid-svg-F8eHn5pSgzeISsIj{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-F8eHn5pSgzeISsIj .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-F8eHn5pSgzeISsIj .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-F8eHn5pSgzeISsIj .error-icon{fill:#552222;}#mermaid-svg-F8eHn5pSgzeISsIj .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-F8eHn5pSgzeISsIj .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-F8eHn5pSgzeISsIj .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-F8eHn5pSgzeISsIj .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-F8eHn5pSgzeISsIj .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-F8eHn5pSgzeISsIj .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-F8eHn5pSgzeISsIj .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-F8eHn5pSgzeISsIj .marker{fill:#333333;stroke:#333333;}#mermaid-svg-F8eHn5pSgzeISsIj .marker.cross{stroke:#333333;}#mermaid-svg-F8eHn5pSgzeISsIj svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-F8eHn5pSgzeISsIj p{margin:0;}#mermaid-svg-F8eHn5pSgzeISsIj .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-F8eHn5pSgzeISsIj .cluster-label text{fill:#333;}#mermaid-svg-F8eHn5pSgzeISsIj .cluster-label span{color:#333;}#mermaid-svg-F8eHn5pSgzeISsIj .cluster-label span p{background-color:transparent;}#mermaid-svg-F8eHn5pSgzeISsIj .label text,#mermaid-svg-F8eHn5pSgzeISsIj span{fill:#333;color:#333;}#mermaid-svg-F8eHn5pSgzeISsIj .node rect,#mermaid-svg-F8eHn5pSgzeISsIj .node circle,#mermaid-svg-F8eHn5pSgzeISsIj .node ellipse,#mermaid-svg-F8eHn5pSgzeISsIj .node polygon,#mermaid-svg-F8eHn5pSgzeISsIj .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-F8eHn5pSgzeISsIj .rough-node .label text,#mermaid-svg-F8eHn5pSgzeISsIj .node .label text,#mermaid-svg-F8eHn5pSgzeISsIj .image-shape .label,#mermaid-svg-F8eHn5pSgzeISsIj .icon-shape .label{text-anchor:middle;}#mermaid-svg-F8eHn5pSgzeISsIj .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-F8eHn5pSgzeISsIj .rough-node .label,#mermaid-svg-F8eHn5pSgzeISsIj .node .label,#mermaid-svg-F8eHn5pSgzeISsIj .image-shape .label,#mermaid-svg-F8eHn5pSgzeISsIj .icon-shape .label{text-align:center;}#mermaid-svg-F8eHn5pSgzeISsIj .node.clickable{cursor:pointer;}#mermaid-svg-F8eHn5pSgzeISsIj .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-F8eHn5pSgzeISsIj .arrowheadPath{fill:#333333;}#mermaid-svg-F8eHn5pSgzeISsIj .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-F8eHn5pSgzeISsIj .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-F8eHn5pSgzeISsIj .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-F8eHn5pSgzeISsIj .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-F8eHn5pSgzeISsIj .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-F8eHn5pSgzeISsIj .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-F8eHn5pSgzeISsIj .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-F8eHn5pSgzeISsIj .cluster text{fill:#333;}#mermaid-svg-F8eHn5pSgzeISsIj .cluster span{color:#333;}#mermaid-svg-F8eHn5pSgzeISsIj div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-F8eHn5pSgzeISsIj .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-F8eHn5pSgzeISsIj rect.text{fill:none;stroke-width:0;}#mermaid-svg-F8eHn5pSgzeISsIj .icon-shape,#mermaid-svg-F8eHn5pSgzeISsIj .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-F8eHn5pSgzeISsIj .icon-shape p,#mermaid-svg-F8eHn5pSgzeISsIj .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-F8eHn5pSgzeISsIj .icon-shape rect,#mermaid-svg-F8eHn5pSgzeISsIj .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-F8eHn5pSgzeISsIj .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-F8eHn5pSgzeISsIj .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-F8eHn5pSgzeISsIj :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

有Combiner

本地聚合

apple,3

1条记录

直接输出

Map

Combiner

Shuffle

Reduce

apple,3

无Combiner

apple,1apple,1apple,1

3条记录

累加

Map

Shuffle

Reduce

apple,3

七、Reduce阶段

7.1 Reducer实现

public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

private IntWritable result = new IntWritable();

@Override
protected void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {

int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result); // 输出最终结果
}
}

7.2 Shuffle过程详解

#mermaid-svg-Mu17BVVifwCBgkS0{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Mu17BVVifwCBgkS0 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Mu17BVVifwCBgkS0 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Mu17BVVifwCBgkS0 .error-icon{fill:#552222;}#mermaid-svg-Mu17BVVifwCBgkS0 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Mu17BVVifwCBgkS0 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Mu17BVVifwCBgkS0 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Mu17BVVifwCBgkS0 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Mu17BVVifwCBgkS0 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Mu17BVVifwCBgkS0 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Mu17BVVifwCBgkS0 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Mu17BVVifwCBgkS0 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Mu17BVVifwCBgkS0 .marker.cross{stroke:#333333;}#mermaid-svg-Mu17BVVifwCBgkS0 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Mu17BVVifwCBgkS0 p{margin:0;}#mermaid-svg-Mu17BVVifwCBgkS0 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-Mu17BVVifwCBgkS0 .cluster-label text{fill:#333;}#mermaid-svg-Mu17BVVifwCBgkS0 .cluster-label span{color:#333;}#mermaid-svg-Mu17BVVifwCBgkS0 .cluster-label span p{background-color:transparent;}#mermaid-svg-Mu17BVVifwCBgkS0 .label text,#mermaid-svg-Mu17BVVifwCBgkS0 span{fill:#333;color:#333;}#mermaid-svg-Mu17BVVifwCBgkS0 .node rect,#mermaid-svg-Mu17BVVifwCBgkS0 .node circle,#mermaid-svg-Mu17BVVifwCBgkS0 .node ellipse,#mermaid-svg-Mu17BVVifwCBgkS0 .node polygon,#mermaid-svg-Mu17BVVifwCBgkS0 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Mu17BVVifwCBgkS0 .rough-node .label text,#mermaid-svg-Mu17BVVifwCBgkS0 .node .label text,#mermaid-svg-Mu17BVVifwCBgkS0 .image-shape .label,#mermaid-svg-Mu17BVVifwCBgkS0 .icon-shape .label{text-anchor:middle;}#mermaid-svg-Mu17BVVifwCBgkS0 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Mu17BVVifwCBgkS0 .rough-node .label,#mermaid-svg-Mu17BVVifwCBgkS0 .node .label,#mermaid-svg-Mu17BVVifwCBgkS0 .image-shape .label,#mermaid-svg-Mu17BVVifwCBgkS0 .icon-shape .label{text-align:center;}#mermaid-svg-Mu17BVVifwCBgkS0 .node.clickable{cursor:pointer;}#mermaid-svg-Mu17BVVifwCBgkS0 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Mu17BVVifwCBgkS0 .arrowheadPath{fill:#333333;}#mermaid-svg-Mu17BVVifwCBgkS0 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Mu17BVVifwCBgkS0 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Mu17BVVifwCBgkS0 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Mu17BVVifwCBgkS0 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Mu17BVVifwCBgkS0 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Mu17BVVifwCBgkS0 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Mu17BVVifwCBgkS0 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Mu17BVVifwCBgkS0 .cluster text{fill:#333;}#mermaid-svg-Mu17BVVifwCBgkS0 .cluster span{color:#333;}#mermaid-svg-Mu17BVVifwCBgkS0 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Mu17BVVifwCBgkS0 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Mu17BVVifwCBgkS0 rect.text{fill:none;stroke-width:0;}#mermaid-svg-Mu17BVVifwCBgkS0 .icon-shape,#mermaid-svg-Mu17BVVifwCBgkS0 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Mu17BVVifwCBgkS0 .icon-shape p,#mermaid-svg-Mu17BVVifwCBgkS0 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Mu17BVVifwCBgkS0 .icon-shape rect,#mermaid-svg-Mu17BVVifwCBgkS0 .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Mu17BVVifwCBgkS0 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Mu17BVVifwCBgkS0 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Mu17BVVifwCBgkS0 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

Shuffle过程

分区

溢写

合并

拉取

合并排序

Map输出

内存缓冲区

本地磁盘多个小文件

一个大文件按分区排序

Reduce

Reducer输入

八、完整WordCount示例

8.1 主类实现

public class WordCount {

public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");

// 设置Jar包
job.setJarByClass(WordCount.class);

// 设置Mapper
job.setMapperClass(WordCountMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);

// 设置Combiner
job.setCombinerClass(WordCountCombiner.class);

// 设置Partitioner
job.setPartitionerClass(HashPartitioner.class);

// 设置Reducer
job.setReducerClass(WordCountReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);

// 设置输入输出路径
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

// 设置Reducer数量
job.setNumReduceTasks(2);

// 提交作业
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}

九、执行流程总结

十、总结

核心要点：

一句话总结：MapReduce通过分而治之的思想，将大数据处理分解为Map（分）和Reduce（合）两个阶段，中间通过Shuffle连接，形成了一个完整的数据处理流水线。