MapReduce常见问题深度解析：数据倾斜与解决方案

MapReduce常见问题深度解析：数据倾斜与解决方案

• • 引言
  • 一、问题现象描述
  • • 1.1 你的理解验证
    • 1.2 问题可视化
  • 二、数据倾斜的常见场景
  • • 2.1 典型场景示例
    • 2.2 代码示例：热点Key问题
  • 三、数据倾斜的定位方法
  • • 3.1 通过JobTracker界面观察
    • 3.2 通过日志分析
  • 四、解决方案一：自定义分区规则
  • • 4.1 重写Partitioner
    • 4.2 复合Key分区
  • 五、解决方案二：加盐（Salting）技术
  • • 5.1 Mapper端加盐
    • 5.2 Reducer端去盐
    • 5.3 加盐效果对比
  • 六、解决方案三：Combiner预处理
  • • 6.1 使用Combiner减少数据量
    • 6.2 Combiner效果对比
  • 七、解决方案四：调整Reducer数量
  • • 7.1 动态调整分区数
  • 八、解决方案五：针对空值的特殊处理
  • • 8.1 空值处理
  • 九、问题排查清单
  • • 9.1 数据倾斜快速诊断
    • 9.2 解决方案选择指南
  • 十、总结

引言

在MapReduce作业运行过程中，数据倾斜是最常见也最棘手的性能问题。它的典型表现是：大部分任务已经完成，但少数几个Reduce任务迟迟无法结束。本文将深入分析这类问题的成因、定位方法以及多种解决方案。

一、问题现象描述

1.1 你的理解验证

“作业中大部分都完成了，但是总有几个reduce一直在运行。这是因为这几个reduce中的处理的数据要远远大于其他的reduce，可能是对键值对任务划分的不均匀造成的数据倾斜。”

✅ 完全正确！ 这就是典型的数据倾斜现象。

1.2 问题可视化

#mermaid-svg-ljctnDUwdhCtRjnn{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ljctnDUwdhCtRjnn .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ljctnDUwdhCtRjnn .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ljctnDUwdhCtRjnn .error-icon{fill:#552222;}#mermaid-svg-ljctnDUwdhCtRjnn .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ljctnDUwdhCtRjnn .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ljctnDUwdhCtRjnn .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ljctnDUwdhCtRjnn .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ljctnDUwdhCtRjnn .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ljctnDUwdhCtRjnn .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ljctnDUwdhCtRjnn .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ljctnDUwdhCtRjnn .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ljctnDUwdhCtRjnn .marker.cross{stroke:#333333;}#mermaid-svg-ljctnDUwdhCtRjnn svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ljctnDUwdhCtRjnn p{margin:0;}#mermaid-svg-ljctnDUwdhCtRjnn .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-ljctnDUwdhCtRjnn .cluster-label text{fill:#333;}#mermaid-svg-ljctnDUwdhCtRjnn .cluster-label span{color:#333;}#mermaid-svg-ljctnDUwdhCtRjnn .cluster-label span p{background-color:transparent;}#mermaid-svg-ljctnDUwdhCtRjnn .label text,#mermaid-svg-ljctnDUwdhCtRjnn span{fill:#333;color:#333;}#mermaid-svg-ljctnDUwdhCtRjnn .node rect,#mermaid-svg-ljctnDUwdhCtRjnn .node circle,#mermaid-svg-ljctnDUwdhCtRjnn .node ellipse,#mermaid-svg-ljctnDUwdhCtRjnn .node polygon,#mermaid-svg-ljctnDUwdhCtRjnn .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-ljctnDUwdhCtRjnn .rough-node .label text,#mermaid-svg-ljctnDUwdhCtRjnn .node .label text,#mermaid-svg-ljctnDUwdhCtRjnn .image-shape .label,#mermaid-svg-ljctnDUwdhCtRjnn .icon-shape .label{text-anchor:middle;}#mermaid-svg-ljctnDUwdhCtRjnn .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-ljctnDUwdhCtRjnn .rough-node .label,#mermaid-svg-ljctnDUwdhCtRjnn .node .label,#mermaid-svg-ljctnDUwdhCtRjnn .image-shape .label,#mermaid-svg-ljctnDUwdhCtRjnn .icon-shape .label{text-align:center;}#mermaid-svg-ljctnDUwdhCtRjnn .node.clickable{cursor:pointer;}#mermaid-svg-ljctnDUwdhCtRjnn .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-ljctnDUwdhCtRjnn .arrowheadPath{fill:#333333;}#mermaid-svg-ljctnDUwdhCtRjnn .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-ljctnDUwdhCtRjnn .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-ljctnDUwdhCtRjnn .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ljctnDUwdhCtRjnn .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-ljctnDUwdhCtRjnn .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ljctnDUwdhCtRjnn .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-ljctnDUwdhCtRjnn .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-ljctnDUwdhCtRjnn .cluster text{fill:#333;}#mermaid-svg-ljctnDUwdhCtRjnn .cluster span{color:#333;}#mermaid-svg-ljctnDUwdhCtRjnn div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-ljctnDUwdhCtRjnn .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-ljctnDUwdhCtRjnn rect.text{fill:none;stroke-width:0;}#mermaid-svg-ljctnDUwdhCtRjnn .icon-shape,#mermaid-svg-ljctnDUwdhCtRjnn .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ljctnDUwdhCtRjnn .icon-shape p,#mermaid-svg-ljctnDUwdhCtRjnn .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-ljctnDUwdhCtRjnn .icon-shape rect,#mermaid-svg-ljctnDUwdhCtRjnn .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ljctnDUwdhCtRjnn .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-ljctnDUwdhCtRjnn .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-ljctnDUwdhCtRjnn :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

数据倾斜情况

1分钟

1分钟

10分钟

Reducer 150万数据

完成

Reducer 250万数据

完成

Reducer 3500万数据

仍在运行

正常情况

2分钟

2分钟

2分钟

Reducer 1100万数据

完成

Reducer 2100万数据

完成

Reducer 3100万数据

完成

二、数据倾斜的常见场景

2.1 典型场景示例

2.2 代码示例：热点Key问题

public class SkewExample {

// 问题场景：统计每个城市的订单量
public static class CityMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

public void map(LongWritable key, Text value, Context context) {
String line = value.toString();
String[] fields = line.split(",");

String city = fields[1]; // 城市字段

// 问题：北京可能有100万订单，其他城市只有几千
context.write(new Text(city), new IntWritable(1));
}
}

// 结果：处理北京的Reducer要处理100万数据，其他只处理几千
}

三、数据倾斜的定位方法

3.1 通过JobTracker界面观察

# 查看作业执行状态
http://jobtracker:50030/jobdetails.jsp?jobid=job_20240214_0001

# 观察Reducer的执行进度
Reducer 1: 100% (完成)
Reducer 2: 100% (完成)
Reducer 3: 100% (完成)
Reducer 4: 35% (长时间卡住) # 数据倾斜！
Reducer 5: 100% (完成)

3.2 通过日志分析

// 在Reducer中记录处理数据量
public static class CounterReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

public void reduce(Text key, Iterable<IntWritable> values, Context context) {
int count = 0;
for (IntWritable val : values) {
count++;
}

// 记录每个Reducer处理的数据量
context.getCounter("ReducerStats", "Key_" + key.toString()).increment(count);
context.write(key, new IntWritable(count));
}
}

// 查看Counters输出
ReducerStats
Key_北京 = 1000000
Key_上海 = 500000
Key_广州 = 300000
Key_深圳 = 200000
Key_其他 = 50000 # 明显倾斜！

四、解决方案一：自定义分区规则

4.1 重写Partitioner

/**
* 方案1：自定义分区器，将热点Key分散到多个Reducer
*/
public class SkewPartitioner extends Partitioner<Text, IntWritable> {

private static final Set<String> HOT_CITIES = new HashSet<>();

static {
HOT_CITIES.add("北京");
HOT_CITIES.add("上海");
HOT_CITIES.add("广州");
HOT_CITIES.add("深圳");
}

@Override
public int getPartition(Text key, IntWritable value, int numPartitions) {
String city = key.toString();

// 1. 如果是热点城市，通过加后缀分散到多个Reducer
if (HOT_CITIES.contains(city)) {
// 为热点城市预留前4个Reducer
int hash = (city + "_" + value.toString()).hashCode();
return Math.abs(hash) % 4; // 分散到0-3号Reducer
}

// 2. 其他城市正常分配
return (city.hashCode() & Integer.MAX_VALUE) % (numPartitions – 4) + 4;
}
}

// 在Job中设置
job.setPartitionerClass(SkewPartitioner.class);

4.2 复合Key分区

/**
* 方案2：使用复合Key，将热点Key拆分
*/
public class CompositeKeyPartitioner extends Partitioner<Text, IntWritable> {

@Override
public int getPartition(Text key, IntWritable value, int numPartitions) {
String keyStr = key.toString();

// 如果是热点Key，提取随机后缀进行分区
if (keyStr.contains("_")) {
String[] parts = keyStr.split("_");
String realKey = parts[0];
String suffix = parts[1];

// 使用真实Key + 后缀决定分区
return (realKey.hashCode() + suffix.hashCode()) % numPartitions;
}

return key.hashCode() % numPartitions;
}
}

五、解决方案二：加盐（Salting）技术

5.1 Mapper端加盐

/**
* 方案3：在Mapper端给热点Key加随机前缀
*/
public class SaltingMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

private static final int SALT_RANGE = 10; // 加10个随机前缀
private static final Set<String> HOT_KEYS = new HashSet<>();

static {
HOT_KEYS.add("北京");
HOT_KEYS.add("上海");
}

public void map(LongWritable key, Text value, Context context) {
String line = value.toString();
String[] fields = line.split(",");
String city = fields[1];
Random random = new Random();

// 如果是热点Key，加随机前缀
if (HOT_KEYS.contains(city)) {
int salt = random.nextInt(SALT_RANGE);
String saltedKey = city + "_" + salt; // 如 "北京_3"
context.write(new Text(saltedKey), new IntWritable(1));
} else {
context.write(new Text(city), new IntWritable(1));
}
}
}

5.2 Reducer端去盐

/**
* 两阶段聚合：先局部聚合，再全局聚合
*/
public class TwoStageReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

// 第一阶段：局部聚合（带盐的Key）
public static class FirstReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

public void reduce(Text key, Iterable<IntWritable> values, Context context) {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}

// 输出带盐的Key的局部结果
context.write(key, new IntWritable(sum));
}
}

// 第二阶段：全局聚合（去盐）
public static class SecondReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

public void reduce(Text key, Iterable<IntWritable> values, Context context) {
// 去掉盐前缀
String originalKey = key.toString().split("_")[0];

int total = 0;
for (IntWritable val : values) {
total += val.get();
}

context.write(new Text(originalKey), new IntWritable(total));
}
}
}

5.3 加盐效果对比

#mermaid-svg-1t7DQknEGeQO7G6P{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-1t7DQknEGeQO7G6P .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-1t7DQknEGeQO7G6P .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-1t7DQknEGeQO7G6P .error-icon{fill:#552222;}#mermaid-svg-1t7DQknEGeQO7G6P .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-1t7DQknEGeQO7G6P .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-1t7DQknEGeQO7G6P .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-1t7DQknEGeQO7G6P .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-1t7DQknEGeQO7G6P .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-1t7DQknEGeQO7G6P .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-1t7DQknEGeQO7G6P .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-1t7DQknEGeQO7G6P .marker{fill:#333333;stroke:#333333;}#mermaid-svg-1t7DQknEGeQO7G6P .marker.cross{stroke:#333333;}#mermaid-svg-1t7DQknEGeQO7G6P svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-1t7DQknEGeQO7G6P p{margin:0;}#mermaid-svg-1t7DQknEGeQO7G6P .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-1t7DQknEGeQO7G6P .cluster-label text{fill:#333;}#mermaid-svg-1t7DQknEGeQO7G6P .cluster-label span{color:#333;}#mermaid-svg-1t7DQknEGeQO7G6P .cluster-label span p{background-color:transparent;}#mermaid-svg-1t7DQknEGeQO7G6P .label text,#mermaid-svg-1t7DQknEGeQO7G6P span{fill:#333;color:#333;}#mermaid-svg-1t7DQknEGeQO7G6P .node rect,#mermaid-svg-1t7DQknEGeQO7G6P .node circle,#mermaid-svg-1t7DQknEGeQO7G6P .node ellipse,#mermaid-svg-1t7DQknEGeQO7G6P .node polygon,#mermaid-svg-1t7DQknEGeQO7G6P .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-1t7DQknEGeQO7G6P .rough-node .label text,#mermaid-svg-1t7DQknEGeQO7G6P .node .label text,#mermaid-svg-1t7DQknEGeQO7G6P .image-shape .label,#mermaid-svg-1t7DQknEGeQO7G6P .icon-shape .label{text-anchor:middle;}#mermaid-svg-1t7DQknEGeQO7G6P .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-1t7DQknEGeQO7G6P .rough-node .label,#mermaid-svg-1t7DQknEGeQO7G6P .node .label,#mermaid-svg-1t7DQknEGeQO7G6P .image-shape .label,#mermaid-svg-1t7DQknEGeQO7G6P .icon-shape .label{text-align:center;}#mermaid-svg-1t7DQknEGeQO7G6P .node.clickable{cursor:pointer;}#mermaid-svg-1t7DQknEGeQO7G6P .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-1t7DQknEGeQO7G6P .arrowheadPath{fill:#333333;}#mermaid-svg-1t7DQknEGeQO7G6P .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-1t7DQknEGeQO7G6P .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-1t7DQknEGeQO7G6P .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-1t7DQknEGeQO7G6P .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-1t7DQknEGeQO7G6P .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-1t7DQknEGeQO7G6P .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-1t7DQknEGeQO7G6P .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-1t7DQknEGeQO7G6P .cluster text{fill:#333;}#mermaid-svg-1t7DQknEGeQO7G6P .cluster span{color:#333;}#mermaid-svg-1t7DQknEGeQO7G6P div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-1t7DQknEGeQO7G6P .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-1t7DQknEGeQO7G6P rect.text{fill:none;stroke-width:0;}#mermaid-svg-1t7DQknEGeQO7G6P .icon-shape,#mermaid-svg-1t7DQknEGeQO7G6P .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-1t7DQknEGeQO7G6P .icon-shape p,#mermaid-svg-1t7DQknEGeQO7G6P .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-1t7DQknEGeQO7G6P .icon-shape rect,#mermaid-svg-1t7DQknEGeQO7G6P .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-1t7DQknEGeQO7G6P .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-1t7DQknEGeQO7G6P .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-1t7DQknEGeQO7G6P :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

加盐后

随机加盐0-9

随机加盐0-9

随机加盐0-9

随机加盐0-9

北京数据100万

Reducer110万

Reducer210万

Reducer310万

…共10个Reducer

加盐前

全到Reducer1

全到Reducer2

分散

北京数据100万条

Reducer1100万

上海数据80万条

Reducer280万

其他数据20万条

其他Reducer各几万

六、解决方案三：Combiner预处理

6.1 使用Combiner减少数据量

/**
* 方案4：使用Combiner进行本地聚合
*/
public class SkewCombiner extends Reducer<Text, IntWritable, Text, IntWritable> {

private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values, Context context) {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);

// Combiner输出局部聚合结果
context.write(key, result);
}
}

// 在Job中设置
job.setCombinerClass(SkewCombiner.class);

6.2 Combiner效果对比

七、解决方案四：调整Reducer数量

7.1 动态调整分区数

/**
* 方案5：增加Reducer数量，减小每个Reducer负担
*/
public class ReducerCountAdjustment {

public static void main(String[] args) {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "Skewed Job");

// 估算数据总量
long totalDataSize = estimateInputSize();

// 根据数据量调整Reducer数量
int numReducers = calculateReducerCount(totalDataSize);

// 通常每个Reducer处理1-2GB数据
job.setNumReduceTasks(numReducers);
}

private static int calculateReducerCount(long dataSize) {
// 每个Reducer期望处理的数据量（1GB）
long targetSizePerReducer = 1024 * 1024 * 1024L;

int reducers = (int) (dataSize / targetSizePerReducer);
return Math.max(1, Math.min(reducers, 100)); // 1-100之间
}
}

八、解决方案五：针对空值的特殊处理

8.1 空值处理

/**
* 方案6：处理空值导致的数据倾斜
*/
public class NullValueHandlingMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

private static final Text NULL_PLACEHOLDER = new Text("__NULL__");

public void map(LongWritable key, Text value, Context context) {
String line = value.toString();
String[] fields = line.split(",");

// 假设分组字段是用户ID
String userId = fields[0];

if (userId == null || userId.isEmpty()) {
// 方案A：给空值加随机后缀分散
Random rand = new Random();
String nullKey = "__NULL_" + rand.nextInt(10);
context.write(new Text(nullKey), new IntWritable(1));

// 方案B：过滤掉空值（如果业务允许）
// return;

// 方案C：赋予默认值
// context.write(NULL_PLACEHOLDER, new IntWritable(1));
} else {
context.write(new Text(userId), new IntWritable(1));
}
}
}

九、问题排查清单

9.1 数据倾斜快速诊断

9.2 解决方案选择指南

#mermaid-svg-N8oLYGSbh3sAvoRU{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-N8oLYGSbh3sAvoRU .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-N8oLYGSbh3sAvoRU .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-N8oLYGSbh3sAvoRU .error-icon{fill:#552222;}#mermaid-svg-N8oLYGSbh3sAvoRU .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-N8oLYGSbh3sAvoRU .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-N8oLYGSbh3sAvoRU .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-N8oLYGSbh3sAvoRU .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-N8oLYGSbh3sAvoRU .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-N8oLYGSbh3sAvoRU .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-N8oLYGSbh3sAvoRU .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-N8oLYGSbh3sAvoRU .marker{fill:#333333;stroke:#333333;}#mermaid-svg-N8oLYGSbh3sAvoRU .marker.cross{stroke:#333333;}#mermaid-svg-N8oLYGSbh3sAvoRU svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-N8oLYGSbh3sAvoRU p{margin:0;}#mermaid-svg-N8oLYGSbh3sAvoRU .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-N8oLYGSbh3sAvoRU .cluster-label text{fill:#333;}#mermaid-svg-N8oLYGSbh3sAvoRU .cluster-label span{color:#333;}#mermaid-svg-N8oLYGSbh3sAvoRU .cluster-label span p{background-color:transparent;}#mermaid-svg-N8oLYGSbh3sAvoRU .label text,#mermaid-svg-N8oLYGSbh3sAvoRU span{fill:#333;color:#333;}#mermaid-svg-N8oLYGSbh3sAvoRU .node rect,#mermaid-svg-N8oLYGSbh3sAvoRU .node circle,#mermaid-svg-N8oLYGSbh3sAvoRU .node ellipse,#mermaid-svg-N8oLYGSbh3sAvoRU .node polygon,#mermaid-svg-N8oLYGSbh3sAvoRU .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-N8oLYGSbh3sAvoRU .rough-node .label text,#mermaid-svg-N8oLYGSbh3sAvoRU .node .label text,#mermaid-svg-N8oLYGSbh3sAvoRU .image-shape .label,#mermaid-svg-N8oLYGSbh3sAvoRU .icon-shape .label{text-anchor:middle;}#mermaid-svg-N8oLYGSbh3sAvoRU .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-N8oLYGSbh3sAvoRU .rough-node .label,#mermaid-svg-N8oLYGSbh3sAvoRU .node .label,#mermaid-svg-N8oLYGSbh3sAvoRU .image-shape .label,#mermaid-svg-N8oLYGSbh3sAvoRU .icon-shape .label{text-align:center;}#mermaid-svg-N8oLYGSbh3sAvoRU .node.clickable{cursor:pointer;}#mermaid-svg-N8oLYGSbh3sAvoRU .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-N8oLYGSbh3sAvoRU .arrowheadPath{fill:#333333;}#mermaid-svg-N8oLYGSbh3sAvoRU .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-N8oLYGSbh3sAvoRU .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-N8oLYGSbh3sAvoRU .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-N8oLYGSbh3sAvoRU .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-N8oLYGSbh3sAvoRU .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-N8oLYGSbh3sAvoRU .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-N8oLYGSbh3sAvoRU .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-N8oLYGSbh3sAvoRU .cluster text{fill:#333;}#mermaid-svg-N8oLYGSbh3sAvoRU .cluster span{color:#333;}#mermaid-svg-N8oLYGSbh3sAvoRU div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-N8oLYGSbh3sAvoRU .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-N8oLYGSbh3sAvoRU rect.text{fill:none;stroke-width:0;}#mermaid-svg-N8oLYGSbh3sAvoRU .icon-shape,#mermaid-svg-N8oLYGSbh3sAvoRU .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-N8oLYGSbh3sAvoRU .icon-shape p,#mermaid-svg-N8oLYGSbh3sAvoRU .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-N8oLYGSbh3sAvoRU .icon-shape rect,#mermaid-svg-N8oLYGSbh3sAvoRU .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-N8oLYGSbh3sAvoRU .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-N8oLYGSbh3sAvoRU .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-N8oLYGSbh3sAvoRU :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

热点Key

空值过多

分区不合理

单个热点Key

多个热点Key

发现数据倾斜

倾斜原因

Key数量

过滤/加盐

自定义Partitioner

加盐技术

分散到多个Reducer

两阶段聚合

自定义分区器

处理完成

十、总结

核心要点：

一句话总结：数据倾斜是MapReduce作业性能的"杀手"，通过合理的数据分区、加盐技术和Combiner预处理，可以有效缓解甚至避免这一问题。