Spark RDD深度解析：概念、创建方式与分区机制

Spark RDD深度解析：概念、创建方式与分区机制

• • 一、RDD是什么？
  • • 1.1 定义解读
    • 1.2 RDD的五大特性
    • 1.3 RDD的形象理解
  • 二、RDD的三种创建方式
  • • 2.1 创建方式总览
    • 2.2 方式一：通过集合创建
    • 2.3 方式二：通过外部数据源创建
    • 2.4 方式三：通过转换创建
  • 三、三种创建方式的分区对比
  • • 3.1 分区策略对比表
    • 3.2 详细说明
    • • 集合创建分区
      • 外部数据源分区
      • 转换创建分区
  • 四、分区数的确定与调整
  • • 4.1 查看分区信息
    • 4.2 分区数的影响
    • 4.3 分区数的选择原则
  • 五、面试高频问题
  • • Q1：RDD是什么？有什么特点？
    • Q2：RDD的创建方式有哪几种？分别怎么分区？
    • Q3：textFile的分区数是怎么确定的？
    • Q4：如何调整RDD的分区数？
    • Q5：分区数设置多少合适？
  • 六、总结
  • • 6.1 RDD创建方式速记
    • 6.2 核心要点

关键词：RDD、弹性分布式数据集、创建方式、分区策略、并行度、数据抽象

在Spark生态系统中，RDD（Resilient Distributed Dataset）是最核心、最基础的数据抽象。理解RDD的本质、创建方式以及分区机制，是掌握Spark编程的第一步，也是面试中的高频考点。

今天，我们将深入剖析RDD的概念内涵、三种创建方式以及各自的分区策略，帮助你全面掌握这一Spark基石。

一、RDD是什么？

1.1 定义解读

RDD（弹性分布式数据集）是Spark中最基本的数据抽象，它代表一个不可变的、可分区的、内部元素可并行计算的集合。

#mermaid-svg-Z8AXlcv3b73hcJMp{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Z8AXlcv3b73hcJMp .error-icon{fill:#552222;}#mermaid-svg-Z8AXlcv3b73hcJMp .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Z8AXlcv3b73hcJMp .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Z8AXlcv3b73hcJMp .marker.cross{stroke:#333333;}#mermaid-svg-Z8AXlcv3b73hcJMp svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Z8AXlcv3b73hcJMp p{margin:0;}#mermaid-svg-Z8AXlcv3b73hcJMp .edge{stroke-width:3;}#mermaid-svg-Z8AXlcv3b73hcJMp .section–1 rect,#mermaid-svg-Z8AXlcv3b73hcJMp .section–1 path,#mermaid-svg-Z8AXlcv3b73hcJMp .section–1 circle,#mermaid-svg-Z8AXlcv3b73hcJMp .section–1 polygon,#mermaid-svg-Z8AXlcv3b73hcJMp .section–1 path{fill:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .section–1 text{fill:#ffffff;}#mermaid-svg-Z8AXlcv3b73hcJMp .node-icon–1{font-size:40px;color:#ffffff;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-edge–1{stroke:hsl(240, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-depth–1{stroke-width:17;}#mermaid-svg-Z8AXlcv3b73hcJMp .section–1 line{stroke:hsl(60, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled circle,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:lightgray;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:#efefef;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-0 rect,#mermaid-svg-Z8AXlcv3b73hcJMp .section-0 path,#mermaid-svg-Z8AXlcv3b73hcJMp .section-0 circle,#mermaid-svg-Z8AXlcv3b73hcJMp .section-0 polygon,#mermaid-svg-Z8AXlcv3b73hcJMp .section-0 path{fill:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-Z8AXlcv3b73hcJMp .section-0 text{fill:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .node-icon-0{font-size:40px;color:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-edge-0{stroke:hsl(60, 100%, 73.5294117647%);}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-depth-0{stroke-width:14;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-0 line{stroke:hsl(240, 100%, 83.5294117647%);stroke-width:3;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled circle,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:lightgray;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:#efefef;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-1 rect,#mermaid-svg-Z8AXlcv3b73hcJMp .section-1 path,#mermaid-svg-Z8AXlcv3b73hcJMp .section-1 circle,#mermaid-svg-Z8AXlcv3b73hcJMp .section-1 polygon,#mermaid-svg-Z8AXlcv3b73hcJMp .section-1 path{fill:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .section-1 text{fill:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .node-icon-1{font-size:40px;color:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-edge-1{stroke:hsl(80, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-depth-1{stroke-width:11;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-1 line{stroke:hsl(260, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled circle,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:lightgray;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:#efefef;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-2 rect,#mermaid-svg-Z8AXlcv3b73hcJMp .section-2 path,#mermaid-svg-Z8AXlcv3b73hcJMp .section-2 circle,#mermaid-svg-Z8AXlcv3b73hcJMp .section-2 polygon,#mermaid-svg-Z8AXlcv3b73hcJMp .section-2 path{fill:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .section-2 text{fill:#ffffff;}#mermaid-svg-Z8AXlcv3b73hcJMp .node-icon-2{font-size:40px;color:#ffffff;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-edge-2{stroke:hsl(270, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-depth-2{stroke-width:8;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-2 line{stroke:hsl(90, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled circle,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:lightgray;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:#efefef;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-3 rect,#mermaid-svg-Z8AXlcv3b73hcJMp .section-3 path,#mermaid-svg-Z8AXlcv3b73hcJMp .section-3 circle,#mermaid-svg-Z8AXlcv3b73hcJMp .section-3 polygon,#mermaid-svg-Z8AXlcv3b73hcJMp .section-3 path{fill:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .section-3 text{fill:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .node-icon-3{font-size:40px;color:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-edge-3{stroke:hsl(300, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-depth-3{stroke-width:5;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-3 line{stroke:hsl(120, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled circle,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:lightgray;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:#efefef;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-4 rect,#mermaid-svg-Z8AXlcv3b73hcJMp .section-4 path,#mermaid-svg-Z8AXlcv3b73hcJMp .section-4 circle,#mermaid-svg-Z8AXlcv3b73hcJMp .section-4 polygon,#mermaid-svg-Z8AXlcv3b73hcJMp .section-4 path{fill:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .section-4 text{fill:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .node-icon-4{font-size:40px;color:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-edge-4{stroke:hsl(330, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-depth-4{stroke-width:2;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-4 line{stroke:hsl(150, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled circle,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:lightgray;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:#efefef;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-5 rect,#mermaid-svg-Z8AXlcv3b73hcJMp .section-5 path,#mermaid-svg-Z8AXlcv3b73hcJMp .section-5 circle,#mermaid-svg-Z8AXlcv3b73hcJMp .section-5 polygon,#mermaid-svg-Z8AXlcv3b73hcJMp .section-5 path{fill:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .section-5 text{fill:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .node-icon-5{font-size:40px;color:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-edge-5{stroke:hsl(0, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-depth-5{stroke-width:-1;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-5 line{stroke:hsl(180, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled circle,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:lightgray;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:#efefef;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-6 rect,#mermaid-svg-Z8AXlcv3b73hcJMp .section-6 path,#mermaid-svg-Z8AXlcv3b73hcJMp .section-6 circle,#mermaid-svg-Z8AXlcv3b73hcJMp .section-6 polygon,#mermaid-svg-Z8AXlcv3b73hcJMp .section-6 path{fill:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .section-6 text{fill:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .node-icon-6{font-size:40px;color:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-edge-6{stroke:hsl(30, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-depth-6{stroke-width:-4;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-6 line{stroke:hsl(210, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled circle,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:lightgray;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:#efefef;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-7 rect,#mermaid-svg-Z8AXlcv3b73hcJMp .section-7 path,#mermaid-svg-Z8AXlcv3b73hcJMp .section-7 circle,#mermaid-svg-Z8AXlcv3b73hcJMp .section-7 polygon,#mermaid-svg-Z8AXlcv3b73hcJMp .section-7 path{fill:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .section-7 text{fill:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .node-icon-7{font-size:40px;color:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-edge-7{stroke:hsl(90, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-depth-7{stroke-width:-7;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-7 line{stroke:hsl(270, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled circle,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:lightgray;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:#efefef;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-8 rect,#mermaid-svg-Z8AXlcv3b73hcJMp .section-8 path,#mermaid-svg-Z8AXlcv3b73hcJMp .section-8 circle,#mermaid-svg-Z8AXlcv3b73hcJMp .section-8 polygon,#mermaid-svg-Z8AXlcv3b73hcJMp .section-8 path{fill:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .section-8 text{fill:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .node-icon-8{font-size:40px;color:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-edge-8{stroke:hsl(150, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-depth-8{stroke-width:-10;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-8 line{stroke:hsl(330, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled circle,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:lightgray;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:#efefef;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-9 rect,#mermaid-svg-Z8AXlcv3b73hcJMp .section-9 path,#mermaid-svg-Z8AXlcv3b73hcJMp .section-9 circle,#mermaid-svg-Z8AXlcv3b73hcJMp .section-9 polygon,#mermaid-svg-Z8AXlcv3b73hcJMp .section-9 path{fill:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .section-9 text{fill:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .node-icon-9{font-size:40px;color:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-edge-9{stroke:hsl(180, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-depth-9{stroke-width:-13;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-9 line{stroke:hsl(0, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled circle,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:lightgray;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:#efefef;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-10 rect,#mermaid-svg-Z8AXlcv3b73hcJMp .section-10 path,#mermaid-svg-Z8AXlcv3b73hcJMp .section-10 circle,#mermaid-svg-Z8AXlcv3b73hcJMp .section-10 polygon,#mermaid-svg-Z8AXlcv3b73hcJMp .section-10 path{fill:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .section-10 text{fill:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .node-icon-10{font-size:40px;color:black;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-edge-10{stroke:hsl(210, 100%, 76.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .edge-depth-10{stroke-width:-16;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-10 line{stroke:hsl(30, 100%, 86.2745098039%);stroke-width:3;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled circle,#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:lightgray;}#mermaid-svg-Z8AXlcv3b73hcJMp .disabled text{fill:#efefef;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-root rect,#mermaid-svg-Z8AXlcv3b73hcJMp .section-root path,#mermaid-svg-Z8AXlcv3b73hcJMp .section-root circle,#mermaid-svg-Z8AXlcv3b73hcJMp .section-root polygon{fill:hsl(240, 100%, 46.2745098039%);}#mermaid-svg-Z8AXlcv3b73hcJMp .section-root text{fill:#ffffff;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-root span{color:#ffffff;}#mermaid-svg-Z8AXlcv3b73hcJMp .section-2 span{color:#ffffff;}#mermaid-svg-Z8AXlcv3b73hcJMp .icon-container{height:100%;display:flex;justify-content:center;align-items:center;}#mermaid-svg-Z8AXlcv3b73hcJMp .edge{fill:none;}#mermaid-svg-Z8AXlcv3b73hcJMp .mindmap-node-label{dy:1em;alignment-baseline:middle;text-anchor:middle;dominant-baseline:middle;text-align:center;}#mermaid-svg-Z8AXlcv3b73hcJMp :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

root（RDD核心特性）

弹性

自动内存和磁盘切换

容错性（血缘关系）

可重算

分布式

数据分布在集群多节点

并行计算

数据集

只读（不可变）

可分区

元素类型任意

1.2 RDD的五大特性

1.3 RDD的形象理解

#mermaid-svg-Q6AKmfue3jptlsAb{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Q6AKmfue3jptlsAb .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Q6AKmfue3jptlsAb .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Q6AKmfue3jptlsAb .error-icon{fill:#552222;}#mermaid-svg-Q6AKmfue3jptlsAb .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Q6AKmfue3jptlsAb .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Q6AKmfue3jptlsAb .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Q6AKmfue3jptlsAb .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Q6AKmfue3jptlsAb .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Q6AKmfue3jptlsAb .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Q6AKmfue3jptlsAb .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Q6AKmfue3jptlsAb .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Q6AKmfue3jptlsAb .marker.cross{stroke:#333333;}#mermaid-svg-Q6AKmfue3jptlsAb svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Q6AKmfue3jptlsAb p{margin:0;}#mermaid-svg-Q6AKmfue3jptlsAb .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-Q6AKmfue3jptlsAb .cluster-label text{fill:#333;}#mermaid-svg-Q6AKmfue3jptlsAb .cluster-label span{color:#333;}#mermaid-svg-Q6AKmfue3jptlsAb .cluster-label span p{background-color:transparent;}#mermaid-svg-Q6AKmfue3jptlsAb .label text,#mermaid-svg-Q6AKmfue3jptlsAb span{fill:#333;color:#333;}#mermaid-svg-Q6AKmfue3jptlsAb .node rect,#mermaid-svg-Q6AKmfue3jptlsAb .node circle,#mermaid-svg-Q6AKmfue3jptlsAb .node ellipse,#mermaid-svg-Q6AKmfue3jptlsAb .node polygon,#mermaid-svg-Q6AKmfue3jptlsAb .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Q6AKmfue3jptlsAb .rough-node .label text,#mermaid-svg-Q6AKmfue3jptlsAb .node .label text,#mermaid-svg-Q6AKmfue3jptlsAb .image-shape .label,#mermaid-svg-Q6AKmfue3jptlsAb .icon-shape .label{text-anchor:middle;}#mermaid-svg-Q6AKmfue3jptlsAb .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Q6AKmfue3jptlsAb .rough-node .label,#mermaid-svg-Q6AKmfue3jptlsAb .node .label,#mermaid-svg-Q6AKmfue3jptlsAb .image-shape .label,#mermaid-svg-Q6AKmfue3jptlsAb .icon-shape .label{text-align:center;}#mermaid-svg-Q6AKmfue3jptlsAb .node.clickable{cursor:pointer;}#mermaid-svg-Q6AKmfue3jptlsAb .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Q6AKmfue3jptlsAb .arrowheadPath{fill:#333333;}#mermaid-svg-Q6AKmfue3jptlsAb .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Q6AKmfue3jptlsAb .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Q6AKmfue3jptlsAb .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Q6AKmfue3jptlsAb .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Q6AKmfue3jptlsAb .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Q6AKmfue3jptlsAb .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Q6AKmfue3jptlsAb .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Q6AKmfue3jptlsAb .cluster text{fill:#333;}#mermaid-svg-Q6AKmfue3jptlsAb .cluster span{color:#333;}#mermaid-svg-Q6AKmfue3jptlsAb div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Q6AKmfue3jptlsAb .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Q6AKmfue3jptlsAb rect.text{fill:none;stroke-width:0;}#mermaid-svg-Q6AKmfue3jptlsAb .icon-shape,#mermaid-svg-Q6AKmfue3jptlsAb .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Q6AKmfue3jptlsAb .icon-shape p,#mermaid-svg-Q6AKmfue3jptlsAb .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Q6AKmfue3jptlsAb .icon-shape rect,#mermaid-svg-Q6AKmfue3jptlsAb .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Q6AKmfue3jptlsAb .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Q6AKmfue3jptlsAb .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Q6AKmfue3jptlsAb :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

RDD内部结构

RDD

分区1Partition 1

分区2Partition 2

分区3Partition 3

分区nPartition n

数据子集

数据子集

数据子集

一句话总结：RDD就是分布在集群多个节点上的、可以并行操作的数据集合。

二、RDD的三种创建方式

2.1 创建方式总览

#mermaid-svg-hBGK7Bzyeppl9I7j{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-hBGK7Bzyeppl9I7j .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-hBGK7Bzyeppl9I7j .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-hBGK7Bzyeppl9I7j .error-icon{fill:#552222;}#mermaid-svg-hBGK7Bzyeppl9I7j .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-hBGK7Bzyeppl9I7j .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-hBGK7Bzyeppl9I7j .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-hBGK7Bzyeppl9I7j .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-hBGK7Bzyeppl9I7j .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-hBGK7Bzyeppl9I7j .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-hBGK7Bzyeppl9I7j .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-hBGK7Bzyeppl9I7j .marker{fill:#333333;stroke:#333333;}#mermaid-svg-hBGK7Bzyeppl9I7j .marker.cross{stroke:#333333;}#mermaid-svg-hBGK7Bzyeppl9I7j svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-hBGK7Bzyeppl9I7j p{margin:0;}#mermaid-svg-hBGK7Bzyeppl9I7j .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-hBGK7Bzyeppl9I7j .cluster-label text{fill:#333;}#mermaid-svg-hBGK7Bzyeppl9I7j .cluster-label span{color:#333;}#mermaid-svg-hBGK7Bzyeppl9I7j .cluster-label span p{background-color:transparent;}#mermaid-svg-hBGK7Bzyeppl9I7j .label text,#mermaid-svg-hBGK7Bzyeppl9I7j span{fill:#333;color:#333;}#mermaid-svg-hBGK7Bzyeppl9I7j .node rect,#mermaid-svg-hBGK7Bzyeppl9I7j .node circle,#mermaid-svg-hBGK7Bzyeppl9I7j .node ellipse,#mermaid-svg-hBGK7Bzyeppl9I7j .node polygon,#mermaid-svg-hBGK7Bzyeppl9I7j .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-hBGK7Bzyeppl9I7j .rough-node .label text,#mermaid-svg-hBGK7Bzyeppl9I7j .node .label text,#mermaid-svg-hBGK7Bzyeppl9I7j .image-shape .label,#mermaid-svg-hBGK7Bzyeppl9I7j .icon-shape .label{text-anchor:middle;}#mermaid-svg-hBGK7Bzyeppl9I7j .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-hBGK7Bzyeppl9I7j .rough-node .label,#mermaid-svg-hBGK7Bzyeppl9I7j .node .label,#mermaid-svg-hBGK7Bzyeppl9I7j .image-shape .label,#mermaid-svg-hBGK7Bzyeppl9I7j .icon-shape .label{text-align:center;}#mermaid-svg-hBGK7Bzyeppl9I7j .node.clickable{cursor:pointer;}#mermaid-svg-hBGK7Bzyeppl9I7j .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-hBGK7Bzyeppl9I7j .arrowheadPath{fill:#333333;}#mermaid-svg-hBGK7Bzyeppl9I7j .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-hBGK7Bzyeppl9I7j .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-hBGK7Bzyeppl9I7j .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hBGK7Bzyeppl9I7j .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-hBGK7Bzyeppl9I7j .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hBGK7Bzyeppl9I7j .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-hBGK7Bzyeppl9I7j .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-hBGK7Bzyeppl9I7j .cluster text{fill:#333;}#mermaid-svg-hBGK7Bzyeppl9I7j .cluster span{color:#333;}#mermaid-svg-hBGK7Bzyeppl9I7j div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-hBGK7Bzyeppl9I7j .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-hBGK7Bzyeppl9I7j rect.text{fill:none;stroke-width:0;}#mermaid-svg-hBGK7Bzyeppl9I7j .icon-shape,#mermaid-svg-hBGK7Bzyeppl9I7j .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hBGK7Bzyeppl9I7j .icon-shape p,#mermaid-svg-hBGK7Bzyeppl9I7j .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-hBGK7Bzyeppl9I7j .icon-shape rect,#mermaid-svg-hBGK7Bzyeppl9I7j .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hBGK7Bzyeppl9I7j .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-hBGK7Bzyeppl9I7j .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-hBGK7Bzyeppl9I7j :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

RDD创建方式

方式一：通过集合创建makeRDD/parallelize

方式二：通过外部数据源创建textFile/hadoopFile

方式三：通过转换创建map/filter等算子

RDD

2.2 方式一：通过集合创建

import org.apache.spark.{SparkConf, SparkContext}

val conf = new SparkConf().setAppName("CreateRDD").setMaster("local[*]")
val sc = new SparkContext(conf)

// 方法1：使用parallelize
val rdd1 = sc.parallelize(Seq(1, 2, 3, 4, 5, 6, 7, 8, 9, 10))
// 或指定分区数
val rdd2 = sc.parallelize(Seq(1, 2, 3, 4, 5), 3) // 指定3个分区

// 方法2：使用makeRDD（底层调用的也是parallelize）
val rdd3 = sc.makeRDD(Seq(1, 2, 3, 4, 5, 6, 7, 8, 9, 10))
val rdd4 = sc.makeRDD(Seq(1, 2, 3, 4, 5), 4) // 指定4个分区

// 查看分区数
println(s"rdd1分区数: ${rdd1.getNumPartitions}")
println(s"rdd2分区数: ${rdd2.getNumPartitions}")

分区策略：

• 默认分区数 = 本地CPU核数（local模式下）或集群配置的spark.default.parallelism
• 指定分区数：可以通过第二个参数显式指定

// 查看默认并行度
val defaultParallelism = sc.defaultParallelism
println(s"默认并行度: $defaultParallelism") // local[*]下等于CPU核数

2.3 方式二：通过外部数据源创建

// 1. 从HDFS读取文件
val rddHdfs = sc.textFile("hdfs://namenode:9000/user/data/input.txt")

// 2. 从本地文件系统读取
val rddLocal = sc.textFile("file:///home/hadoop/data/words.txt")

// 3. 读取多个文件（可以使用通配符）
val rddMultiple = sc.textFile("/data/logs/2024-01-15/part-*.log")

// 4. 读取整个目录
val rddDir = sc.textFile("/data/input/")

// 5. 指定最小分区数
val rddWithMinPartitions = sc.textFile("/data/input.txt", 5) // 最小5个分区

// 6. 读取其他格式
val sequenceRDD = sc.sequenceFile[String, Int]("/data/seq/")
val hadoopRDD = sc.hadoopFile[Text, Text, InputFormat]("/data/hadoop/")

分区策略：

#mermaid-svg-gd57Lk4D7P8Nh3L6{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-gd57Lk4D7P8Nh3L6 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .error-icon{fill:#552222;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .marker.cross{stroke:#333333;}#mermaid-svg-gd57Lk4D7P8Nh3L6 svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-gd57Lk4D7P8Nh3L6 p{margin:0;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .cluster-label text{fill:#333;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .cluster-label span{color:#333;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .cluster-label span p{background-color:transparent;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .label text,#mermaid-svg-gd57Lk4D7P8Nh3L6 span{fill:#333;color:#333;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .node rect,#mermaid-svg-gd57Lk4D7P8Nh3L6 .node circle,#mermaid-svg-gd57Lk4D7P8Nh3L6 .node ellipse,#mermaid-svg-gd57Lk4D7P8Nh3L6 .node polygon,#mermaid-svg-gd57Lk4D7P8Nh3L6 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .rough-node .label text,#mermaid-svg-gd57Lk4D7P8Nh3L6 .node .label text,#mermaid-svg-gd57Lk4D7P8Nh3L6 .image-shape .label,#mermaid-svg-gd57Lk4D7P8Nh3L6 .icon-shape .label{text-anchor:middle;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .rough-node .label,#mermaid-svg-gd57Lk4D7P8Nh3L6 .node .label,#mermaid-svg-gd57Lk4D7P8Nh3L6 .image-shape .label,#mermaid-svg-gd57Lk4D7P8Nh3L6 .icon-shape .label{text-align:center;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .node.clickable{cursor:pointer;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .arrowheadPath{fill:#333333;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-gd57Lk4D7P8Nh3L6 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-gd57Lk4D7P8Nh3L6 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-gd57Lk4D7P8Nh3L6 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .cluster text{fill:#333;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .cluster span{color:#333;}#mermaid-svg-gd57Lk4D7P8Nh3L6 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-gd57Lk4D7P8Nh3L6 rect.text{fill:none;stroke-width:0;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .icon-shape,#mermaid-svg-gd57Lk4D7P8Nh3L6 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .icon-shape p,#mermaid-svg-gd57Lk4D7P8Nh3L6 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .icon-shape rect,#mermaid-svg-gd57Lk4D7P8Nh3L6 .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-gd57Lk4D7P8Nh3L6 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-gd57Lk4D7P8Nh3L6 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-gd57Lk4D7P8Nh3L6 :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

HDFS文件分区

文件如 260MB

HDFS分块1128MB

HDFS分块2128MB

HDFS分块34MB

RDD分区1

RDD分区2

RDD分区3

分区数计算公式：

• 基础分区数 = HDFS文件块数（每个块至少一个分区）
• 最小分区数：可以通过minPartitions参数指定
• 最终分区数 = max(文件块数, minPartitions)

// 深入理解分区计算
// 假设文件大小 260MB，HDFS块大小 128MB
// 文件块数 = ceil(260/128) = 3个块

// 不指定minPartitions
val rdd1 = sc.textFile("/data/260mb.txt")
// 分区数 = 3

// 指定minPartitions=5
val rdd2 = sc.textFile("/data/260mb.txt", 5)
// 分区数 = max(3, 5) = 5
// 但实际HadoopInputFormat会尝试将数据尽量均匀分到5个分区

2.4 方式三：通过转换创建

// 从已有RDD通过转换算子创建新RDD
val originalRDD = sc.parallelize(Seq(1, 2, 3, 4, 5, 6, 7, 8), 4)
println(s"原始RDD分区数: ${originalRDD.getNumPartitions}") // 4

// map转换：分区数不变
val mapRDD = originalRDD.map(_ * 2)
println(s"map后RDD分区数: ${mapRDD.getNumPartitions}") // 4

// filter转换：分区数不变，但可能某些分区数据变少或为空
val filterRDD = originalRDD.filter(_ > 3)
println(s"filter后RDD分区数: ${filterRDD.getNumPartitions}") // 4

// 宽依赖转换：repartition可以改变分区数
val repartitionRDD = originalRDD.repartition(6)
println(s"repartition后分区数: ${repartitionRDD.getNumPartitions}") // 6

// 宽依赖转换：reduceByKey等聚合操作
val pairRDD = sc.parallelize(Seq(("a", 1), ("b", 2), ("a", 3), ("c", 4)), 2)
val reduceRDD = pairRDD.reduceByKey(_ + _)
println(s"reduceByKey后分区数: ${reduceRDD.getNumPartitions}")
// 默认等于父RDD的最大分区数，或通过参数指定

分区策略：

#mermaid-svg-KprGEXzToesjRGag{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-KprGEXzToesjRGag .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-KprGEXzToesjRGag .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-KprGEXzToesjRGag .error-icon{fill:#552222;}#mermaid-svg-KprGEXzToesjRGag .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-KprGEXzToesjRGag .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-KprGEXzToesjRGag .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-KprGEXzToesjRGag .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-KprGEXzToesjRGag .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-KprGEXzToesjRGag .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-KprGEXzToesjRGag .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-KprGEXzToesjRGag .marker{fill:#333333;stroke:#333333;}#mermaid-svg-KprGEXzToesjRGag .marker.cross{stroke:#333333;}#mermaid-svg-KprGEXzToesjRGag svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-KprGEXzToesjRGag p{margin:0;}#mermaid-svg-KprGEXzToesjRGag .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-KprGEXzToesjRGag .cluster-label text{fill:#333;}#mermaid-svg-KprGEXzToesjRGag .cluster-label span{color:#333;}#mermaid-svg-KprGEXzToesjRGag .cluster-label span p{background-color:transparent;}#mermaid-svg-KprGEXzToesjRGag .label text,#mermaid-svg-KprGEXzToesjRGag span{fill:#333;color:#333;}#mermaid-svg-KprGEXzToesjRGag .node rect,#mermaid-svg-KprGEXzToesjRGag .node circle,#mermaid-svg-KprGEXzToesjRGag .node ellipse,#mermaid-svg-KprGEXzToesjRGag .node polygon,#mermaid-svg-KprGEXzToesjRGag .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-KprGEXzToesjRGag .rough-node .label text,#mermaid-svg-KprGEXzToesjRGag .node .label text,#mermaid-svg-KprGEXzToesjRGag .image-shape .label,#mermaid-svg-KprGEXzToesjRGag .icon-shape .label{text-anchor:middle;}#mermaid-svg-KprGEXzToesjRGag .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-KprGEXzToesjRGag .rough-node .label,#mermaid-svg-KprGEXzToesjRGag .node .label,#mermaid-svg-KprGEXzToesjRGag .image-shape .label,#mermaid-svg-KprGEXzToesjRGag .icon-shape .label{text-align:center;}#mermaid-svg-KprGEXzToesjRGag .node.clickable{cursor:pointer;}#mermaid-svg-KprGEXzToesjRGag .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-KprGEXzToesjRGag .arrowheadPath{fill:#333333;}#mermaid-svg-KprGEXzToesjRGag .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-KprGEXzToesjRGag .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-KprGEXzToesjRGag .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-KprGEXzToesjRGag .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-KprGEXzToesjRGag .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-KprGEXzToesjRGag .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-KprGEXzToesjRGag .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-KprGEXzToesjRGag .cluster text{fill:#333;}#mermaid-svg-KprGEXzToesjRGag .cluster span{color:#333;}#mermaid-svg-KprGEXzToesjRGag div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-KprGEXzToesjRGag .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-KprGEXzToesjRGag rect.text{fill:none;stroke-width:0;}#mermaid-svg-KprGEXzToesjRGag .icon-shape,#mermaid-svg-KprGEXzToesjRGag .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-KprGEXzToesjRGag .icon-shape p,#mermaid-svg-KprGEXzToesjRGag .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-KprGEXzToesjRGag .icon-shape rect,#mermaid-svg-KprGEXzToesjRGag .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-KprGEXzToesjRGag .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-KprGEXzToesjRGag .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-KprGEXzToesjRGag :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

宽依赖转换

Shuffle

Shuffle

Shuffle

Shuffle

父RDD分区1

子RDD分区1

子RDD分区2

父RDD分区2

分区数可能变化默认等于父RDD最大分区数或通过参数指定

窄依赖转换

一对一

一对一

一对一

父RDD分区1

子RDD分区1

父RDD分区2

子RDD分区2

父RDD分区3

子RDD分区3

分区数不变

三、三种创建方式的分区对比

3.1 分区策略对比表

3.2 详细说明

集合创建分区

// 默认并行度的确定
val conf = new SparkConf()
.setAppName("Demo")
.setMaster("local[5]") // 显式指定5个核
// 或 .setMaster("local[*]") // 使用所有可用核

// 默认并行度 =
// 1. 如果显式设置了spark.default.parallelism，使用该值
// 2. 否则在local模式下等于CPU核数
// 3. 在集群模式下等于所有executor核数总和

外部数据源分区

// textFile的minPartitions参数只是"建议"最小值
// HadoopInputFormat有自己的分区逻辑

// 文件分区规则
// 对于不可分割的文件格式（如gzip），整个文件作为一个分区
// 对于可分割格式，每个HDFS块至少一个分区
// minPartitions会触发更细粒度的分割，但不会小于HDFS块边界

转换创建分区

// 窄依赖：分区数不变
val narrowRDD = rdd.map(x => x) // 分区数不变

// 宽依赖：分区数可配置
val rdd1 = rdd.reduceByKey(_ + _) // 使用默认并行度
val rdd2 = rdd.reduceByKey(_ + _, 8) // 指定8个分区

// repartition：可以任意改变分区数
val rdd3 = rdd.repartition(10) // 增加分区
val rdd4 = rdd.coalesce(2) // 减少分区（默认不Shuffle）

四、分区数的确定与调整

4.1 查看分区信息

// 查看分区数
val partitionCount = rdd.getNumPartitions
println(s"分区数: $partitionCount")

// 查看每个分区的数据量
val partitionSizes = rdd.mapPartitions(iter => Iterator(iter.length)).collect()
println(s"各分区大小: ${partitionSizes.mkString(", ")}")

// 查看分区器（仅对PairRDD有效）
val partitioner = rdd.partitioner
println(s"分区器: $partitioner")

4.2 分区数的影响

4.3 分区数的选择原则

// 经验公式
// 分区数 = 总数据量 / 目标分区大小

// 目标分区大小建议：128MB – 256MB
// 例如：100GB数据，目标分区大小128MB
// 分区数 = 100 * 1024 / 128 ≈ 800

// 同时考虑CPU核数
// 分区数应该至少为executor核数的2-3倍
val executorCores = 100 // 假设总核数100
val recommendedPartitions = executorCores * 3 // 300

// 最终分区数 = min(数据量计算, 核数倍数)

五、面试高频问题

Q1：RDD是什么？有什么特点？

答：RDD是弹性分布式数据集，核心特点：

• 弹性：内存不足时可写到磁盘，容错性强
• 分布式：数据分布在集群多个节点
• 不可变：只能通过转换生成新RDD
• 可分区：支持并行计算
• 惰性求值：只有action才真正计算

Q2：RDD的创建方式有哪几种？分别怎么分区？

答：三种创建方式：

Q3：textFile的分区数是怎么确定的？

答：取决于：

• HDFS文件块数（每个块至少一个分区）
• minPartitions参数（建议最小值）
• 文件是否可分割（gzip等不可分割格式整个文件一个分区）
• 最终分区数 = max(文件块数, minPartitions)

Q4：如何调整RDD的分区数？

答：

• repartition(n)：增加或减少分区，会Shuffle
• coalesce(n, shuffle=false)：减少分区，默认不Shuffle
• 宽依赖算子指定分区数：reduceByKey(func, numPartitions)

Q5：分区数设置多少合适？

答：综合考虑：

• 每个分区处理128MB-256MB数据为宜
• 分区数至少为executor总核数的2-3倍
• 避免分区过多导致调度开销过大
• 避免分区过少导致OOM或资源浪费

六、总结

6.1 RDD创建方式速记

#mermaid-svg-wwodA5TUlA6mLEFi{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-wwodA5TUlA6mLEFi .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-wwodA5TUlA6mLEFi .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-wwodA5TUlA6mLEFi .error-icon{fill:#552222;}#mermaid-svg-wwodA5TUlA6mLEFi .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-wwodA5TUlA6mLEFi .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-wwodA5TUlA6mLEFi .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-wwodA5TUlA6mLEFi .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-wwodA5TUlA6mLEFi .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-wwodA5TUlA6mLEFi .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-wwodA5TUlA6mLEFi .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-wwodA5TUlA6mLEFi .marker{fill:#333333;stroke:#333333;}#mermaid-svg-wwodA5TUlA6mLEFi .marker.cross{stroke:#333333;}#mermaid-svg-wwodA5TUlA6mLEFi svg{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-wwodA5TUlA6mLEFi p{margin:0;}#mermaid-svg-wwodA5TUlA6mLEFi .label{font-family:\”trebuchet ms\”,verdana,arial,sans-serif;color:#333;}#mermaid-svg-wwodA5TUlA6mLEFi .cluster-label text{fill:#333;}#mermaid-svg-wwodA5TUlA6mLEFi .cluster-label span{color:#333;}#mermaid-svg-wwodA5TUlA6mLEFi .cluster-label span p{background-color:transparent;}#mermaid-svg-wwodA5TUlA6mLEFi .label text,#mermaid-svg-wwodA5TUlA6mLEFi span{fill:#333;color:#333;}#mermaid-svg-wwodA5TUlA6mLEFi .node rect,#mermaid-svg-wwodA5TUlA6mLEFi .node circle,#mermaid-svg-wwodA5TUlA6mLEFi .node ellipse,#mermaid-svg-wwodA5TUlA6mLEFi .node polygon,#mermaid-svg-wwodA5TUlA6mLEFi .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-wwodA5TUlA6mLEFi .rough-node .label text,#mermaid-svg-wwodA5TUlA6mLEFi .node .label text,#mermaid-svg-wwodA5TUlA6mLEFi .image-shape .label,#mermaid-svg-wwodA5TUlA6mLEFi .icon-shape .label{text-anchor:middle;}#mermaid-svg-wwodA5TUlA6mLEFi .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-wwodA5TUlA6mLEFi .rough-node .label,#mermaid-svg-wwodA5TUlA6mLEFi .node .label,#mermaid-svg-wwodA5TUlA6mLEFi .image-shape .label,#mermaid-svg-wwodA5TUlA6mLEFi .icon-shape .label{text-align:center;}#mermaid-svg-wwodA5TUlA6mLEFi .node.clickable{cursor:pointer;}#mermaid-svg-wwodA5TUlA6mLEFi .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-wwodA5TUlA6mLEFi .arrowheadPath{fill:#333333;}#mermaid-svg-wwodA5TUlA6mLEFi .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-wwodA5TUlA6mLEFi .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-wwodA5TUlA6mLEFi .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-wwodA5TUlA6mLEFi .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-wwodA5TUlA6mLEFi .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-wwodA5TUlA6mLEFi .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-wwodA5TUlA6mLEFi .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-wwodA5TUlA6mLEFi .cluster text{fill:#333;}#mermaid-svg-wwodA5TUlA6mLEFi .cluster span{color:#333;}#mermaid-svg-wwodA5TUlA6mLEFi div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:\”trebuchet ms\”,verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-wwodA5TUlA6mLEFi .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-wwodA5TUlA6mLEFi rect.text{fill:none;stroke-width:0;}#mermaid-svg-wwodA5TUlA6mLEFi .icon-shape,#mermaid-svg-wwodA5TUlA6mLEFi .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-wwodA5TUlA6mLEFi .icon-shape p,#mermaid-svg-wwodA5TUlA6mLEFi .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-wwodA5TUlA6mLEFi .icon-shape rect,#mermaid-svg-wwodA5TUlA6mLEFi .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-wwodA5TUlA6mLEFi .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-wwodA5TUlA6mLEFi .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-wwodA5TUlA6mLEFi :root{–mermaid-font-family:\”trebuchet ms\”,verdana,arial,sans-serif;}

RDD创建

parallelize/makeRDD

textFile/hadoopFile

map/filter等

分区

分区

分区

集合创建

RDD

外部数据源

转换创建

默认=CPU核数

默认=HDFS块数

窄依赖：继承宽依赖：可指定

6.2 核心要点

RDD是Spark的基石，创建方式是入口，分区策略定并行

掌握RDD的创建和分区机制，你就能从源头控制Spark作业的并行度和性能，为后续的优化打下坚实基础！

思考题：在Spark 2.0+中，DataFrame/Dataset也是基于RDD实现的，它们的分区机制和RDD有什么异同？欢迎在评论区讨论！