spark的shuffle原理及调优

1、shuffle原理：
当使?reduceByKey、groupByKey、sortByKey、countByKey、join、cogroup等操作的时候，会发?shuffle操作。

spark的shuffle实现：在DAG阶段以shuffle为界，划分stage，上游stage 做map task，每个map task将计算结果数据分成多份，每?份对应到下游stage的每个partition中，并将其临时写到磁盘，该过程就叫做shuffle write；下游stage叫做reduce task，每个reduce task通过?络拉取指定分区结果数据，该过程叫做shuffle read， 最后完成reduce的业务逻辑。

2、shuffle的优化：
1）提前过滤数据：在 join、groupBy 等宽依赖操作之前，尽早使用 filter 减少数据量

2）合理使用广播变量：当一个大表和一个小表进行 join 时，将小表（通常在几百 MB 以内）用 broadcast 广播到每个 Executor，可以将 SortMergeJoin 降级为 BroadcastHashJoin，彻底避免 Shuffle。

3）提前进行聚合：在 join 之前，先对数据进行聚合缩减数据量。

4）调整 Shuffle 分区数，让每个分区的数据量控制在 200MB 到 1GB 左右，并且让分区数尽量与 CPU 核心数匹配，以提高并行度。

5）动态调整 Shuffle 和执行计划

动态合并 Shuffle 分区：自动将 Shuffle 产生的小文件合并成合适的尺寸。

动态调整 Join 策略：如果运行时发现表的大小适合广播，会自动将 SortMergeJoin 转换为 BroadcastHashJoin。
6）减少 Shuffle 数据溢出，调整 Shuffle 写数据落盘的缓冲区大小

spark.shuffle.file.buffer：Map 端 Shuffle 写数据时的缓冲区，默认 32KB。增大该值（例如 64KB 或 128KB）可以减少磁盘 I/O 次数。

spark.file.transferTo：默认为 true，使用零拷贝（NIO）传输数据。如果遇到兼容性问题或 NIO 表现不佳，可以设为 false 回退到传统 I/O
https://www.douban.com/topic/479226600/
https://www.douban.com/topic/479226600/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479226600/
https://m.douban.com/topic/479226600/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479226472/
https://www.douban.com/topic/479226472/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479226472/
https://m.douban.com/topic/479226472/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479226353/
https://www.douban.com/topic/479226353/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479226353/
https://m.douban.com/topic/479226353/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479226227/
https://www.douban.com/topic/479226227/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479226227/
https://m.douban.com/topic/479226227/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479226108/
https://www.douban.com/topic/479226108/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479226108/
https://m.douban.com/topic/479226108/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479226023/
https://www.douban.com/topic/479226023/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479226023/
https://m.douban.com/topic/479226023/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479225933/
https://www.douban.com/topic/479225933/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479225933/
https://m.douban.com/topic/479225933/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479225812/
https://www.douban.com/topic/479225812/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479225812/
https://m.douban.com/topic/479225812/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479225695/
https://www.douban.com/topic/479225695/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479225695/
https://m.douban.com/topic/479225695/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479225590/
https://www.douban.com/topic/479225590/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479225590/
https://m.douban.com/topic/479225590/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479225484/
https://www.douban.com/topic/479225484/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479225484/
https://m.douban.com/topic/479225484/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479225377/
https://www.douban.com/topic/479225377/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479225377/
https://m.douban.com/topic/479225377/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479225259/
https://www.douban.com/topic/479225259/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479225259/
https://m.douban.com/topic/479225259/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479225157/
https://www.douban.com/topic/479225157/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479225157/
https://m.douban.com/topic/479225157/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479225033/
https://www.douban.com/topic/479225033/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479225033/
https://m.douban.com/topic/479225033/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479224918/
https://www.douban.com/topic/479224918/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479224918/
https://m.douban.com/topic/479224918/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479224786/
https://www.douban.com/topic/479224786/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479224786/
https://m.douban.com/topic/479224786/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479224662/
https://www.douban.com/topic/479224662/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479224662/
https://m.douban.com/topic/479224662/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479224543/
https://www.douban.com/topic/479224543/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479224543/
https://m.douban.com/topic/479224543/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479224435/
https://www.douban.com/topic/479224435/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479224435/
https://m.douban.com/topic/479224435/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479224363/
https://www.douban.com/topic/479224363/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479224363/
https://m.douban.com/topic/479224363/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479224298/
https://www.douban.com/topic/479224298/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479224298/
https://m.douban.com/topic/479224298/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479224196/
https://www.douban.com/topic/479224196/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479224196/
https://m.douban.com/topic/479224196/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479224078/
https://www.douban.com/topic/479224078/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479224078/
https://m.douban.com/topic/479224078/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479223967/
https://www.douban.com/topic/479223967/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479223967/
https://m.douban.com/topic/479223967/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479223870/
https://www.douban.com/topic/479223870/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479223870/
https://m.douban.com/topic/479223870/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479223749/
https://www.douban.com/topic/479223749/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479223749/
https://m.douban.com/topic/479223749/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479223639/
https://www.douban.com/topic/479223639/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479223639/
https://m.douban.com/topic/479223639/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479223527/
https://www.douban.com/topic/479223527/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479223527/
https://m.douban.com/topic/479223527/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479223421/
https://www.douban.com/topic/479223421/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479223421/
https://m.douban.com/topic/479223421/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479223318/
https://www.douban.com/topic/479223318/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479223318/
https://m.douban.com/topic/479223318/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479223208/
https://www.douban.com/topic/479223208/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479223208/
https://m.douban.com/topic/479223208/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479223095/
https://www.douban.com/topic/479223095/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479223095/
https://m.douban.com/topic/479223095/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479222981/
https://www.douban.com/topic/479222981/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479222981/
https://m.douban.com/topic/479222981/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479222911/
https://www.douban.com/topic/479222911/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479222911/
https://m.douban.com/topic/479222911/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479222835/
https://www.douban.com/topic/479222835/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479222835/
https://m.douban.com/topic/479222835/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479222727/
https://www.douban.com/topic/479222727/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479222727/
https://m.douban.com/topic/479222727/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479222624/
https://www.douban.com/topic/479222624/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479222624/
https://m.douban.com/topic/479222624/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479222565/
https://www.douban.com/topic/479222565/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479222565/
https://m.douban.com/topic/479222565/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479222495/
https://www.douban.com/topic/479222495/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479222495/
https://m.douban.com/topic/479222495/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479221138/
https://www.douban.com/topic/479221138/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479221138/
https://m.douban.com/topic/479221138/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479221092/
https://www.douban.com/topic/479221092/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479221092/
https://m.douban.com/topic/479221092/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479221054/
https://www.douban.com/topic/479221054/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479221054/
https://m.douban.com/topic/479221054/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479220919/
https://www.douban.com/topic/479220919/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479220919/
https://m.douban.com/topic/479220919/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479220842/
https://www.douban.com/topic/479220842/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479220842/
https://m.douban.com/topic/479220842/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479220639/
https://www.douban.com/topic/479220639/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479220639/
https://m.douban.com/topic/479220639/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479220513/
https://www.douban.com/topic/479220513/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479220513/
https://m.douban.com/topic/479220513/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479220369/
https://www.douban.com/topic/479220369/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479220369/
https://m.douban.com/topic/479220369/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479218402/
https://www.douban.com/topic/479218402/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479218402/
https://m.douban.com/topic/479218402/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479218320/
https://www.douban.com/topic/479218320/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479218320/
https://m.douban.com/topic/479218320/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479218232/
https://www.douban.com/topic/479218232/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479218232/
https://m.douban.com/topic/479218232/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479218089/
https://www.douban.com/topic/479218089/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479218089/
https://m.douban.com/topic/479218089/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479217916/
https://www.douban.com/topic/479217916/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479217916/
https://m.douban.com/topic/479217916/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479217796/
https://www.douban.com/topic/479217796/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479217796/
https://m.douban.com/topic/479217796/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479216132/
https://www.douban.com/topic/479216132/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479216132/
https://m.douban.com/topic/479216132/?_spm_id=Mjk0MDA1NTIx
https://www.douban.com/topic/479197105/
https://www.douban.com/topic/479197105/?_spm_id=Mjk0MDA1NTIx
https://m.douban.com/topic/479197105/
https://m.douban.com/topic/479197105/?_spm_id=Mjk0MDA1NTIx