KafkaUtils.createDStream vs createDirectStream:Spark Streaming整合Kafka的两种方式
-
- 引言
- 一、核心区别概览
- 二、Receiver方式 (createDStream) 详解
-
- 2.1 工作原理图
- 2.2 代码示例
- 2.3 关键配置
- 三、Direct方式 (createDirectStream) 详解
-
- 3.1 工作原理图
- 3.2 代码示例
- 四、核心区别深度对比
-
- 4.1 数据接收与存储
- 4.2 并行度控制
- 4.3 容错与语义保证
- 五、优缺点对比
-
- 5.1 Receiver方式
- 5.2 Direct方式
- 六、偏移量管理对比
-
- 6.1 Receiver方式偏移量管理
- 6.2 Direct方式偏移量管理
- 七、性能对比
- 八、选型建议
-
- 8.1 决策树
- 8.2 场景推荐
- 九、生产环境配置示例
-
- 9.1 Receiver方式配置
- 9.2 Direct方式配置
- 十、总结
|
🌺The Begin🌺点点关注,收藏不迷路🌺 |
引言
在Spark Streaming与Kafka整合的历史演进中,出现过两种主要的API:基于Receiver的createDStream和基于Direct的createDirectStream。理解这两者的区别,对于设计可靠、高效的实时数据处理系统至关重要。本文将深入对比两种方式的原理、优缺点及适用场景。
一、核心区别概览
| 引入版本 | Spark 1.0+ | Spark 1.3+ |
| 数据接收 | Receiver线程持续接收 | Driver触发时直接拉取 |
| 存储方式 | Executor内存 + WAL | 无中间存储 |
| 偏移量管理 | Zookeeper | Spark Checkpoint/外部系统 |
| 语义保证 | At-Least-Once | Exactly-Once |
| 并行度 | 由Receiver数量决定 | 与Kafka分区数一致 |
二、Receiver方式 (createDStream) 详解
2.1 工作原理图
#mermaid-svg-gV550fkZXCe4jwl1{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-gV550fkZXCe4jwl1 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-gV550fkZXCe4jwl1 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-gV550fkZXCe4jwl1 .error-icon{fill:#552222;}#mermaid-svg-gV550fkZXCe4jwl1 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-gV550fkZXCe4jwl1 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-gV550fkZXCe4jwl1 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-gV550fkZXCe4jwl1 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-gV550fkZXCe4jwl1 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-gV550fkZXCe4jwl1 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-gV550fkZXCe4jwl1 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-gV550fkZXCe4jwl1 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-gV550fkZXCe4jwl1 .marker.cross{stroke:#333333;}#mermaid-svg-gV550fkZXCe4jwl1 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-gV550fkZXCe4jwl1 p{margin:0;}#mermaid-svg-gV550fkZXCe4jwl1 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-gV550fkZXCe4jwl1 .cluster-label text{fill:#333;}#mermaid-svg-gV550fkZXCe4jwl1 .cluster-label span{color:#333;}#mermaid-svg-gV550fkZXCe4jwl1 .cluster-label span p{background-color:transparent;}#mermaid-svg-gV550fkZXCe4jwl1 .label text,#mermaid-svg-gV550fkZXCe4jwl1 span{fill:#333;color:#333;}#mermaid-svg-gV550fkZXCe4jwl1 .node rect,#mermaid-svg-gV550fkZXCe4jwl1 .node circle,#mermaid-svg-gV550fkZXCe4jwl1 .node ellipse,#mermaid-svg-gV550fkZXCe4jwl1 .node polygon,#mermaid-svg-gV550fkZXCe4jwl1 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-gV550fkZXCe4jwl1 .rough-node .label text,#mermaid-svg-gV550fkZXCe4jwl1 .node .label text,#mermaid-svg-gV550fkZXCe4jwl1 .image-shape .label,#mermaid-svg-gV550fkZXCe4jwl1 .icon-shape .label{text-anchor:middle;}#mermaid-svg-gV550fkZXCe4jwl1 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-gV550fkZXCe4jwl1 .rough-node .label,#mermaid-svg-gV550fkZXCe4jwl1 .node .label,#mermaid-svg-gV550fkZXCe4jwl1 .image-shape .label,#mermaid-svg-gV550fkZXCe4jwl1 .icon-shape .label{text-align:center;}#mermaid-svg-gV550fkZXCe4jwl1 .node.clickable{cursor:pointer;}#mermaid-svg-gV550fkZXCe4jwl1 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-gV550fkZXCe4jwl1 .arrowheadPath{fill:#333333;}#mermaid-svg-gV550fkZXCe4jwl1 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-gV550fkZXCe4jwl1 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-gV550fkZXCe4jwl1 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-gV550fkZXCe4jwl1 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-gV550fkZXCe4jwl1 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-gV550fkZXCe4jwl1 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-gV550fkZXCe4jwl1 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-gV550fkZXCe4jwl1 .cluster text{fill:#333;}#mermaid-svg-gV550fkZXCe4jwl1 .cluster span{color:#333;}#mermaid-svg-gV550fkZXCe4jwl1 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-gV550fkZXCe4jwl1 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-gV550fkZXCe4jwl1 rect.text{fill:none;stroke-width:0;}#mermaid-svg-gV550fkZXCe4jwl1 .icon-shape,#mermaid-svg-gV550fkZXCe4jwl1 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-gV550fkZXCe4jwl1 .icon-shape p,#mermaid-svg-gV550fkZXCe4jwl1 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-gV550fkZXCe4jwl1 .icon-shape rect,#mermaid-svg-gV550fkZXCe4jwl1 .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-gV550fkZXCe4jwl1 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-gV550fkZXCe4jwl1 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-gV550fkZXCe4jwl1 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
Driver
Spark Executor 2
Spark Executor 1
Kafka集群
更新偏移量
更新偏移量
Partition 0
Partition 1
Partition 2
Receiver线程消费Partition 0&1
BlockManager内存缓存
WAL预写日志HDFS
Receiver线程消费Partition 2
BlockManager内存缓存
WAL预写日志HDFS
ReceiverTracker
Zookeeper偏移量存储
2.2 代码示例
// Receiver方式(Kafka 0.8 API)
import org.apache.spark.streaming.kafka._
import org.apache.spark.storage.StorageLevel
// 启用WAL(可选)
ssc.checkpoint("hdfs://namenode/checkpoint")
System.setProperty("spark.streaming.receiver.writeAheadLog.enable", "true")
val kafkaStream = KafkaUtils.createStream(
ssc,
"zookeeper:2181", // Zookeeper连接
"spark-streaming-group", // Consumer组
Map("topic1" -> 1, "topic2" -> 1), // 主题和并行度
StorageLevel.MEMORY_AND_DISK_SER_2 // 存储级别
)
// 处理数据
kafkaStream.map(record => (record._1, record._2))
.print()
2.3 关键配置
// 启用WAL(写前日志)
sparkConf.set("spark.streaming.receiver.writeAheadLog.enable", "true")
// 存储级别(双副本+序列化+内存磁盘)
StorageLevel.MEMORY_AND_DISK_SER_2
// 背压控制
sparkConf.set("spark.streaming.backpressure.enabled", "true")
sparkConf.set("spark.streaming.kafka.maxRatePerPartition", "1000")
三、Direct方式 (createDirectStream) 详解
3.1 工作原理图
#mermaid-svg-nGzPbOtvYQpcYMUh{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-nGzPbOtvYQpcYMUh .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-nGzPbOtvYQpcYMUh .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-nGzPbOtvYQpcYMUh .error-icon{fill:#552222;}#mermaid-svg-nGzPbOtvYQpcYMUh .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-nGzPbOtvYQpcYMUh .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-nGzPbOtvYQpcYMUh .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-nGzPbOtvYQpcYMUh .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-nGzPbOtvYQpcYMUh .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-nGzPbOtvYQpcYMUh .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-nGzPbOtvYQpcYMUh .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-nGzPbOtvYQpcYMUh .marker{fill:#333333;stroke:#333333;}#mermaid-svg-nGzPbOtvYQpcYMUh .marker.cross{stroke:#333333;}#mermaid-svg-nGzPbOtvYQpcYMUh svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-nGzPbOtvYQpcYMUh p{margin:0;}#mermaid-svg-nGzPbOtvYQpcYMUh .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-nGzPbOtvYQpcYMUh .cluster-label text{fill:#333;}#mermaid-svg-nGzPbOtvYQpcYMUh .cluster-label span{color:#333;}#mermaid-svg-nGzPbOtvYQpcYMUh .cluster-label span p{background-color:transparent;}#mermaid-svg-nGzPbOtvYQpcYMUh .label text,#mermaid-svg-nGzPbOtvYQpcYMUh span{fill:#333;color:#333;}#mermaid-svg-nGzPbOtvYQpcYMUh .node rect,#mermaid-svg-nGzPbOtvYQpcYMUh .node circle,#mermaid-svg-nGzPbOtvYQpcYMUh .node ellipse,#mermaid-svg-nGzPbOtvYQpcYMUh .node polygon,#mermaid-svg-nGzPbOtvYQpcYMUh .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-nGzPbOtvYQpcYMUh .rough-node .label text,#mermaid-svg-nGzPbOtvYQpcYMUh .node .label text,#mermaid-svg-nGzPbOtvYQpcYMUh .image-shape .label,#mermaid-svg-nGzPbOtvYQpcYMUh .icon-shape .label{text-anchor:middle;}#mermaid-svg-nGzPbOtvYQpcYMUh .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-nGzPbOtvYQpcYMUh .rough-node .label,#mermaid-svg-nGzPbOtvYQpcYMUh .node .label,#mermaid-svg-nGzPbOtvYQpcYMUh .image-shape .label,#mermaid-svg-nGzPbOtvYQpcYMUh .icon-shape .label{text-align:center;}#mermaid-svg-nGzPbOtvYQpcYMUh .node.clickable{cursor:pointer;}#mermaid-svg-nGzPbOtvYQpcYMUh .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-nGzPbOtvYQpcYMUh .arrowheadPath{fill:#333333;}#mermaid-svg-nGzPbOtvYQpcYMUh .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-nGzPbOtvYQpcYMUh .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-nGzPbOtvYQpcYMUh .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-nGzPbOtvYQpcYMUh .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-nGzPbOtvYQpcYMUh .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-nGzPbOtvYQpcYMUh .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-nGzPbOtvYQpcYMUh .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-nGzPbOtvYQpcYMUh .cluster text{fill:#333;}#mermaid-svg-nGzPbOtvYQpcYMUh .cluster span{color:#333;}#mermaid-svg-nGzPbOtvYQpcYMUh div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-nGzPbOtvYQpcYMUh .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-nGzPbOtvYQpcYMUh rect.text{fill:none;stroke-width:0;}#mermaid-svg-nGzPbOtvYQpcYMUh .icon-shape,#mermaid-svg-nGzPbOtvYQpcYMUh .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-nGzPbOtvYQpcYMUh .icon-shape p,#mermaid-svg-nGzPbOtvYQpcYMUh .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-nGzPbOtvYQpcYMUh .icon-shape rect,#mermaid-svg-nGzPbOtvYQpcYMUh .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-nGzPbOtvYQpcYMUh .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-nGzPbOtvYQpcYMUh .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-nGzPbOtvYQpcYMUh :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
Executor 2
Executor 1
Kafka集群
Driver
偏移量管理
1. 获取偏移量
1. 获取偏移量
1. 获取偏移量
2. 分配任务
2. 分配任务
2. 分配任务
3. 直接拉取
3. 直接拉取
3. 直接拉取
4. 处理完成
4. 处理完成
4. 处理完成
5. 提交偏移量
Driver
偏移量获取查询Kafka
偏移量存储Checkpoint/外部
Partition 0
Partition 1
Partition 2
Task 1读取Partition 0
Task 2读取Partition 1
Task 3读取Partition 2
3.2 代码示例
// Direct方式(Kafka 0.10+ API)
import org.apache.spark.streaming.kafka010._
import org.apache.kafka.common.serialization.StringDeserializer
val kafkaParams = Map[String, Object](
"bootstrap.servers" -> "broker1:9092,broker2:9092",
"key.deserializer" -> classOf[StringDeserializer],
"value.deserializer" -> classOf[StringDeserializer],
"group.id" -> "spark-direct-demo",
"auto.offset.reset" -> "latest",
"enable.auto.commit" -> (false: java.lang.Boolean) // 手动提交
)
val topics = Array("topic1", "topic2")
val stream = KafkaUtils.createDirectStream[String, String](
ssc,
LocationStrategies.PreferConsistent, // 均匀分布
ConsumerStrategies.Subscribe[String, String](topics, kafkaParams)
)
// 处理数据
stream.foreachRDD { rdd =>
// 获取偏移量
val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
// 业务处理
rdd.map(record => (record.key(), record.value()))
.foreach(processRecord)
// 手动提交偏移量
stream.asInstanceOf[CanCommitOffsets].commitAsync(offsetRanges)
}
四、核心区别深度对比
4.1 数据接收与存储
| 数据接收 | Receiver线程持续接收 | Task直接拉取 |
| 中间存储 | Executor内存+序列化 | 无中间存储 |
| 数据备份 | WAL + 双副本 | 不备份 |
| 内存压力 | 高 | 低 |
4.2 并行度控制
// Receiver方式:并行度由Receiver数量决定
val receiverStream = KafkaUtils.createStream(
ssc,
zkQuorum,
groupId,
Map("topic1" -> 3) // 3个Receiver,3个并行度
)
// Direct方式:并行度由Kafka分区数决定
// 如果topic1有10个分区,自动有10个并行度
val directStream = KafkaUtils.createDirectStream(
ssc,
PreferConsistent,
Subscribe(Array("topic1"), kafkaParams)
)
4.3 容错与语义保证
#mermaid-svg-yodPNHbvqQIonRns{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-yodPNHbvqQIonRns .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-yodPNHbvqQIonRns .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-yodPNHbvqQIonRns .error-icon{fill:#552222;}#mermaid-svg-yodPNHbvqQIonRns .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-yodPNHbvqQIonRns .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-yodPNHbvqQIonRns .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-yodPNHbvqQIonRns .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-yodPNHbvqQIonRns .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-yodPNHbvqQIonRns .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-yodPNHbvqQIonRns .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-yodPNHbvqQIonRns .marker{fill:#333333;stroke:#333333;}#mermaid-svg-yodPNHbvqQIonRns .marker.cross{stroke:#333333;}#mermaid-svg-yodPNHbvqQIonRns svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-yodPNHbvqQIonRns p{margin:0;}#mermaid-svg-yodPNHbvqQIonRns .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-yodPNHbvqQIonRns .cluster-label text{fill:#333;}#mermaid-svg-yodPNHbvqQIonRns .cluster-label span{color:#333;}#mermaid-svg-yodPNHbvqQIonRns .cluster-label span p{background-color:transparent;}#mermaid-svg-yodPNHbvqQIonRns .label text,#mermaid-svg-yodPNHbvqQIonRns span{fill:#333;color:#333;}#mermaid-svg-yodPNHbvqQIonRns .node rect,#mermaid-svg-yodPNHbvqQIonRns .node circle,#mermaid-svg-yodPNHbvqQIonRns .node ellipse,#mermaid-svg-yodPNHbvqQIonRns .node polygon,#mermaid-svg-yodPNHbvqQIonRns .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-yodPNHbvqQIonRns .rough-node .label text,#mermaid-svg-yodPNHbvqQIonRns .node .label text,#mermaid-svg-yodPNHbvqQIonRns .image-shape .label,#mermaid-svg-yodPNHbvqQIonRns .icon-shape .label{text-anchor:middle;}#mermaid-svg-yodPNHbvqQIonRns .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-yodPNHbvqQIonRns .rough-node .label,#mermaid-svg-yodPNHbvqQIonRns .node .label,#mermaid-svg-yodPNHbvqQIonRns .image-shape .label,#mermaid-svg-yodPNHbvqQIonRns .icon-shape .label{text-align:center;}#mermaid-svg-yodPNHbvqQIonRns .node.clickable{cursor:pointer;}#mermaid-svg-yodPNHbvqQIonRns .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-yodPNHbvqQIonRns .arrowheadPath{fill:#333333;}#mermaid-svg-yodPNHbvqQIonRns .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-yodPNHbvqQIonRns .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-yodPNHbvqQIonRns .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-yodPNHbvqQIonRns .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-yodPNHbvqQIonRns .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-yodPNHbvqQIonRns .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-yodPNHbvqQIonRns .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-yodPNHbvqQIonRns .cluster text{fill:#333;}#mermaid-svg-yodPNHbvqQIonRns .cluster span{color:#333;}#mermaid-svg-yodPNHbvqQIonRns div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-yodPNHbvqQIonRns .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-yodPNHbvqQIonRns rect.text{fill:none;stroke-width:0;}#mermaid-svg-yodPNHbvqQIonRns .icon-shape,#mermaid-svg-yodPNHbvqQIonRns .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-yodPNHbvqQIonRns .icon-shape p,#mermaid-svg-yodPNHbvqQIonRns .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-yodPNHbvqQIonRns .icon-shape rect,#mermaid-svg-yodPNHbvqQIonRns .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-yodPNHbvqQIonRns .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-yodPNHbvqQIonRns .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-yodPNHbvqQIonRns :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
Direct方式 – Exactly-Once
从未提交偏移量开始
获取偏移量范围
处理数据
提交偏移量同事务
故障恢复
Receiver方式 – At-Least-Once
重启后重读WAL
接收数据
写入WAL
更新Zookeeper偏移量
处理数据
Driver故障
五、优缺点对比
5.1 Receiver方式
| 实现简单,API友好 | 需要额外内存存储数据 |
| 支持WAL保证数据不丢 | At-Least-Once可能重复 |
| 与旧版Kafka兼容 | Receiver可能成为瓶颈 |
| 内置背压支持 | 数据延迟较大 |
5.2 Direct方式
| Exactly-Once语义 | 需要手动管理偏移量 |
| 并行度与分区一致 | 不支持Kafka 0.8 |
| 无需额外内存存储 | 复杂度略高 |
| 低延迟 | 需要外部系统存储偏移量 |
六、偏移量管理对比
6.1 Receiver方式偏移量管理
// Receiver方式:偏移量存储在Zookeeper
// 自动提交,无需手动处理
val stream = KafkaUtils.createStream(ssc, zkQuorum, groupId, topics)
// 偏移量存储在Zookeeper路径:
// /consumers/[groupId]/offsets/[topic]/[partition]
6.2 Direct方式偏移量管理
// Direct方式:需要手动管理偏移量
// 方案1:提交到Kafka
stream.foreachRDD { rdd =>
val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
// 业务处理
// 提交到Kafka
stream.asInstanceOf[CanCommitOffsets].commitAsync(offsetRanges)
}
// 方案2:存储到外部系统(MySQL/Redis/HBase)
stream.foreachRDD { rdd =>
val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
// 开启事务
beginTransaction()
try {
processData(rdd)
saveOffsetsToHBase(offsetRanges) // 同事务
commitTransaction()
} catch {
case e: Exception => rollbackTransaction()
}
}
// 从HBase恢复偏移量
def getStoredOffsets(topics: Array[String], groupId: String): Map[TopicPartition, Long] = {
// 从HBase读取已提交偏移量
}
七、性能对比
| 最大吞吐 | 10-50 MB/s/node | 50-200 MB/s/node |
| 处理延迟 | 几秒~几分钟 | 毫秒~秒级 |
| 资源消耗 | 高(内存+磁盘) | 低 |
| 扩展性 | 受Receiver限制 | 随分区数扩展 |
八、选型建议
8.1 决策树
#mermaid-svg-LYJfnfx04ch10x6C{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-LYJfnfx04ch10x6C .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-LYJfnfx04ch10x6C .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-LYJfnfx04ch10x6C .error-icon{fill:#552222;}#mermaid-svg-LYJfnfx04ch10x6C .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-LYJfnfx04ch10x6C .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-LYJfnfx04ch10x6C .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-LYJfnfx04ch10x6C .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-LYJfnfx04ch10x6C .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-LYJfnfx04ch10x6C .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-LYJfnfx04ch10x6C .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-LYJfnfx04ch10x6C .marker{fill:#333333;stroke:#333333;}#mermaid-svg-LYJfnfx04ch10x6C .marker.cross{stroke:#333333;}#mermaid-svg-LYJfnfx04ch10x6C svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-LYJfnfx04ch10x6C p{margin:0;}#mermaid-svg-LYJfnfx04ch10x6C .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-LYJfnfx04ch10x6C .cluster-label text{fill:#333;}#mermaid-svg-LYJfnfx04ch10x6C .cluster-label span{color:#333;}#mermaid-svg-LYJfnfx04ch10x6C .cluster-label span p{background-color:transparent;}#mermaid-svg-LYJfnfx04ch10x6C .label text,#mermaid-svg-LYJfnfx04ch10x6C span{fill:#333;color:#333;}#mermaid-svg-LYJfnfx04ch10x6C .node rect,#mermaid-svg-LYJfnfx04ch10x6C .node circle,#mermaid-svg-LYJfnfx04ch10x6C .node ellipse,#mermaid-svg-LYJfnfx04ch10x6C .node polygon,#mermaid-svg-LYJfnfx04ch10x6C .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-LYJfnfx04ch10x6C .rough-node .label text,#mermaid-svg-LYJfnfx04ch10x6C .node .label text,#mermaid-svg-LYJfnfx04ch10x6C .image-shape .label,#mermaid-svg-LYJfnfx04ch10x6C .icon-shape .label{text-anchor:middle;}#mermaid-svg-LYJfnfx04ch10x6C .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-LYJfnfx04ch10x6C .rough-node .label,#mermaid-svg-LYJfnfx04ch10x6C .node .label,#mermaid-svg-LYJfnfx04ch10x6C .image-shape .label,#mermaid-svg-LYJfnfx04ch10x6C .icon-shape .label{text-align:center;}#mermaid-svg-LYJfnfx04ch10x6C .node.clickable{cursor:pointer;}#mermaid-svg-LYJfnfx04ch10x6C .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-LYJfnfx04ch10x6C .arrowheadPath{fill:#333333;}#mermaid-svg-LYJfnfx04ch10x6C .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-LYJfnfx04ch10x6C .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-LYJfnfx04ch10x6C .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-LYJfnfx04ch10x6C .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-LYJfnfx04ch10x6C .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-LYJfnfx04ch10x6C .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-LYJfnfx04ch10x6C .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-LYJfnfx04ch10x6C .cluster text{fill:#333;}#mermaid-svg-LYJfnfx04ch10x6C .cluster span{color:#333;}#mermaid-svg-LYJfnfx04ch10x6C div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-LYJfnfx04ch10x6C .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-LYJfnfx04ch10x6C rect.text{fill:none;stroke-width:0;}#mermaid-svg-LYJfnfx04ch10x6C .icon-shape,#mermaid-svg-LYJfnfx04ch10x6C .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-LYJfnfx04ch10x6C .icon-shape p,#mermaid-svg-LYJfnfx04ch10x6C .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-LYJfnfx04ch10x6C .icon-shape rect,#mermaid-svg-LYJfnfx04ch10x6C .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-LYJfnfx04ch10x6C .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-LYJfnfx04ch10x6C .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-LYJfnfx04ch10x6C :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}
<0.8
>=0.10
Exactly-Once
At-Least-Once
内存充足
追求低延迟
选择Kafka整合方式
Kafka版本
Receiver方式
语义要求
Direct方式
资源情况
Receiver方式启用WAL
Direct方式
8.2 场景推荐
| 金融交易 | Direct | 需要Exactly-Once |
| 日志收集 | Direct | 高吞吐,精确一次 |
| 老系统升级 | Receiver | 兼容Kafka 0.8 |
| 实时风控 | Direct | 低延迟要求 |
| 数据湖摄入 | Direct | 避免数据重复 |
九、生产环境配置示例
9.1 Receiver方式配置
val conf = new SparkConf()
.set("spark.streaming.receiver.writeAheadLog.enable", "true")
.set("spark.streaming.receiver.maxRate", "5000")
.set("spark.streaming.backpressure.enabled", "true")
.set("spark.streaming.backpressure.initialRate", "1000")
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
ssc.checkpoint("hdfs://namenode/checkpoint")
val stream = KafkaUtils.createStream(
ssc,
"zk1:2181,zk2:2181/consumer",
"group1",
Map("topic1" -> 3),
StorageLevel.MEMORY_AND_DISK_SER_2
)
9.2 Direct方式配置
val kafkaParams = Map[String, Object](
"bootstrap.servers" -> "broker1:9092,broker2:9092",
"group.id" -> "group1",
"enable.auto.commit" -> (false: java.lang.Boolean),
"auto.offset.reset" -> "earliest",
"max.poll.records" -> "1000",
"session.timeout.ms" -> "30000",
"heartbeat.interval.ms" -> "3000"
)
val stream = KafkaUtils.createDirectStream(
ssc,
PreferConsistent,
Subscribe(Array("topic1"), kafkaParams)
)
// Exactly-Once处理
stream.foreachRDD { rdd =>
val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
// 业务处理(确保幂等)
rdd.foreachPartition { partition =>
val connection = createHBaseConnection()
partition.foreach { record =>
connection.saveWithRowKey(record.key(), record.value())
}
connection.close()
}
// 提交偏移量
stream.asInstanceOf[CanCommitOffsets].commitAsync(offsetRanges)
}
十、总结
| 核心原理 | Receiver接收+缓存+WAL | Driver调度+Task直连 |
| 数据流 | Kafka → Receiver → BlockManager → Task | Kafka → Task |
| 偏移量管理 | Zookeeper | Spark/外部系统 |
| 语义保证 | At-Least-Once | Exactly-Once |
| 并行度 | 由Receiver数量决定 | 与Kafka分区数一致 |
| 适用版本 | Kafka 0.8+ | Kafka 0.10+ |
| 生产推荐 | 逐步淘汰 | 标准方案 |
一句话总结:
- Receiver方式:先收后处理,有WAL保障,但可能重复
- Direct方式:直接拉取处理,精确一次,推荐使用
随着Kafka和Spark的版本演进,Direct方式已经成为事实标准,提供更好的性能、精确的语义保证和更简洁的架构。

|
🌺The End🌺点点关注,收藏不迷路🌺 |


