存储系统按照抽象级别分类，分为三种：文件存储、对象存储、块存储。此处我们不讨论块存储，只讨论文件存储与对象存储。
文件存储是我们在日常生活中最熟悉的存储方式。它将数据组织成树状结构（目录/文件夹）。每个文件都位于特定的路径中，拥有文件名、大小、创建日期等有限的元数据。 典型的代表有NTFS、ext4、xfs、NTFS等。 访问协议有NFS、SMB/CIFS、POSIX接口。 文件存储的优势在于遍历目录速度快、支持随机写操作，POSIX接口使用广泛，无需客户端进行开发适配。
而对象存储是一种扁平化的存储架构。数据不再以层级结构存储，而是被视为一个“对象”，放到一个“桶”中进行管理。每个对象包含数据、元数据和全局唯一的key来构成。 典型的代表有Amazon S3, Google Cloud Storage, Tencent Cloud Object Storage, Ali OSS, ceph radosgw等。 访问现已为RESTful API(基于HTTP/HTTPS)。 对象存储的优势在于无限可扩展性、低成本、高可用性与持久性，便于网络访问。因此一般用于镜像仓库的存储端、海量文件的备份归档等。
AI训练存储选型的演进路线
第一阶段：单机直连时代
早期的深度学习数据集较小，模型训练通常在单台服务器或单张GPU卡上完成。此时直接将数据存储在训练机器的本地NVMe SSD/HDD上。
其优势在于IO延迟最低，吞吐量极高，也就是“数据离计算最近”。缺点也很明显：
数据孤岛：多台机器无法共享数据，数据拷贝（scp/rsync)及其耗时。
容量受限：本地磁盘容量有限。
数据安全：数据没有冗余，机器或盘挂了，数据可能就丢了。
第二阶段：传统共享存储时代（NFS/NAS）
到后来开始团队协作，训练规模也变大，则需要多机多卡分布式训练，同时出于使用方便，大家希望像操作本地文件一样操作共享数据。
此时开始以NFS（Network File System)挂载传统的NAS存储阵列来使用。
其优势在于POSIX兼容，不需要修改PyTorch/TensorFlow代码，直接读写文件路径，共享也十分便利，所有节点看到的目录结构一致。缺点则在于元数据性能瓶颈，NFS在处理海量小文件的open,lookup操作时，元数据服务器可能被瞬间打死。另外，所有计算节点抢占同一个NAS的出口带宽，也可能导致GPU等待IO，造成GPU无法维持高使用率。
第三阶段：大数据融合时代（HDFS）
Google等大厂入局，Hadoop诞生。数据存储在HDFS上，通过一些软件层面的优化将大量小图片打包成大文件，变成“顺序读”，训练代码通过API读取。这样就能够极大地利用HDFS的设计初衷：吞吐量极高、大规模顺序读，同时还能利用现成的大数据基础设施。
但同样有缺点：
生态割裂：PyTorch对HDFS支持不如TensorFlow友好。
随机访问差：对于需要频繁Shuffle(随机打乱)的数据集，HDFS性能不佳。
Java开销：HDFS客户端通常较重，占用CPU资源
第四阶段：高性能并行文件系统（HPC/Parallel FS)
此时的AI领域进入超算时代，模型越来越大，对低延迟和超高带宽的要求极高。
此时的方案是借用超算（HPC）领域的Lustre或GPFS（IMB Spectrum Scale)。优点在于专为高并发设计，可以轻松喂饱数千张GPU，支持POSIX，且元数据性能极强。缺点则在于贵，运维难（Lustre等系统内合级调优十分困难，一旦集群崩溃，恢复十分困难），扩容困难，很难像云存储那样弹性伸缩。
第五阶段：存算分离与分层架构（Object Storage + Cache)
当前这个时代即云原生时代，数据集达到PB级别，成本成为了核心考量。对象存储最便宜且容量无限，但性能（尤其是元数据）不够好。 此时针对元数据有了若干解决方案：
https://avg.163.com/topic/detail/11096335
https://avg.163.com/topic/detail/11096329
https://avg.163.com/topic/detail/11096320
https://avg.163.com/topic/detail/11096308
https://avg.163.com/topic/detail/11096307
https://avg.163.com/topic/detail/11096304
https://avg.163.com/topic/detail/11096287
https://avg.163.com/topic/detail/11096273
https://avg.163.com/topic/detail/11096268
https://avg.163.com/topic/detail/11096263
https://avg.163.com/topic/detail/11096251
https://avg.163.com/topic/detail/11096241
https://avg.163.com/topic/detail/11096230
https://avg.163.com/topic/detail/11096228
https://avg.163.com/topic/detail/11096218
https://avg.163.com/topic/detail/11096214
https://avg.163.com/topic/detail/11096188
https://avg.163.com/topic/detail/11096178
https://avg.163.com/topic/detail/11096160
https://avg.163.com/topic/detail/11096156
https://avg.163.com/topic/detail/11096151
https://avg.163.com/topic/detail/11096152
https://avg.163.com/topic/detail/11096144
https://avg.163.com/topic/detail/11096136
https://avg.163.com/topic/detail/11096133
https://avg.163.com/topic/detail/11096100
https://avg.163.com/topic/detail/11096076
https://avg.163.com/topic/detail/11096072
https://avg.163.com/topic/detail/11096051
https://avg.163.com/topic/detail/11096049
https://avg.163.com/topic/detail/11096034
https://avg.163.com/topic/detail/11096033
https://avg.163.com/topic/detail/11096024
https://avg.163.com/topic/detail/11096012
https://avg.163.com/topic/detail/11095973
https://avg.163.com/topic/detail/11095962
https://avg.163.com/topic/detail/11095967
https://avg.163.com/topic/detail/11095963
https://avg.163.com/topic/detail/11095959
https://avg.163.com/topic/detail/11095534
https://avg.163.com/topic/detail/11095747
https://avg.163.com/topic/detail/11095944
https://avg.163.com/topic/detail/11095937
https://avg.163.com/topic/detail/11095921
https://avg.163.com/topic/detail/11095904
https://avg.163.com/topic/detail/11095867
https://avg.163.com/topic/detail/11095853
https://avg.163.com/topic/detail/11095855
https://avg.163.com/topic/detail/11095836
https://avg.163.com/topic/detail/11095830
https://avg.163.com/topic/detail/11095817
https://avg.163.com/topic/detail/11095803
https://avg.163.com/topic/detail/11095796
https://avg.163.com/topic/detail/11095787
https://avg.163.com/topic/detail/11095777
https://avg.163.com/topic/detail/11095763
https://avg.163.com/topic/detail/11095747
https://avg.163.com/topic/detail/11095723
https://avg.163.com/topic/detail/11095706
https://avg.163.com/topic/detail/11095683
https://avg.163.com/topic/detail/11095662
https://avg.163.com/topic/detail/11095655
https://avg.163.com/topic/detail/11095643
https://avg.163.com/topic/detail/11095634
https://avg.163.com/topic/detail/11095628
https://avg.163.com/topic/detail/11095617
https://avg.163.com/topic/detail/11095572
https://avg.163.com/topic/detail/11095569
https://avg.163.com/topic/detail/11095563
https://avg.163.com/topic/detail/11095540
https://avg.163.com/topic/detail/11095534
https://avg.163.com/topic/detail/11095523
https://avg.163.com/topic/detail/11095514
https://avg.163.com/topic/detail/11095504
https://avg.163.com/topic/detail/11095490
https://avg.163.com/topic/detail/11095482
https://avg.163.com/topic/detail/11094487
https://avg.163.com/topic/detail/11095445
https://avg.163.com/topic/detail/11095431
https://avg.163.com/topic/detail/11095422
https://avg.163.com/topic/detail/11095414
https://avg.163.com/topic/detail/11095398
https://avg.163.com/topic/detail/11095382
https://avg.163.com/topic/detail/11095359
https://avg.163.com/topic/detail/11095350
https://avg.163.com/topic/detail/11095335
https://avg.163.com/topic/detail/11095317
https://avg.163.com/topic/detail/11095308
https://avg.163.com/topic/detail/11095289
https://avg.163.com/topic/detail/11095274
https://avg.163.com/topic/detail/11094831
https://avg.163.com/topic/detail/11095245
https://avg.163.com/topic/detail/11095228
https://avg.163.com/topic/detail/11095210
https://avg.163.com/topic/detail/11095212
https://avg.163.com/topic/detail/11095211
https://avg.163.com/topic/detail/11095209
https://avg.163.com/topic/detail/11095208
https://avg.163.com/topic/detail/11095195
https://avg.163.com/topic/detail/11095187
https://avg.163.com/topic/detail/11095167
https://avg.163.com/topic/detail/11095159
https://avg.163.com/topic/detail/11095153
https://avg.163.com/topic/detail/11095131
https://avg.163.com/topic/detail/11095125
https://avg.163.com/topic/detail/11095105
https://avg.163.com/topic/detail/11095102
https://avg.163.com/topic/detail/11095088
https://avg.163.com/topic/detail/11095068
https://avg.163.com/topic/detail/11095051
https://avg.163.com/topic/detail/11095046
https://avg.163.com/topic/detail/11095000
https://avg.163.com/topic/detail/11094990
https://avg.163.com/topic/detail/11094983
https://avg.163.com/topic/detail/11094955
https://avg.163.com/topic/detail/11094925
https://avg.163.com/topic/detail/11094916
https://avg.163.com/topic/detail/11094919
https://avg.163.com/topic/detail/11094902
https://avg.163.com/topic/detail/11094897
https://avg.163.com/topic/detail/11094870
https://avg.163.com/topic/detail/11094852
https://avg.163.com/topic/detail/11094846
https://avg.163.com/topic/detail/11094840
https://avg.163.com/topic/detail/11094739
https://avg.163.com/topic/detail/11094811
https://avg.163.com/topic/detail/11094792
https://avg.163.com/topic/detail/11094781
https://avg.163.com/topic/detail/11094777
https://avg.163.com/topic/detail/11094773
https://avg.163.com/topic/detail/11094739
https://avg.163.com/topic/detail/11094720
https://avg.163.com/topic/detail/11094700
https://avg.163.com/topic/detail/11094216
https://avg.163.com/topic/detail/11094690
https://avg.163.com/topic/detail/11094689
https://avg.163.com/topic/detail/11094680
https://avg.163.com/topic/detail/11094663
https://avg.163.com/topic/detail/11094665
https://avg.163.com/topic/detail/11093784
https://avg.163.com/topic/detail/11094626
https://avg.163.com/topic/detail/11094606
https://avg.163.com/topic/detail/11094574
https://avg.163.com/topic/detail/11094569
https://avg.163.com/topic/detail/11094555
https://avg.163.com/topic/detail/11094158
https://avg.163.com/topic/detail/11094529
https://avg.163.com/topic/detail/11093811
https://avg.163.com/topic/detail/11094487
https://avg.163.com/topic/detail/11094447
https://avg.163.com/topic/detail/11094423
https://avg.163.com/topic/detail/11094419
https://avg.163.com/topic/detail/11094418
https://avg.163.com/topic/detail/11094412
https://avg.163.com/topic/detail/11094413
https://avg.163.com/topic/detail/11094410
https://avg.163.com/topic/detail/11094396
https://avg.163.com/topic/detail/11094347
https://avg.163.com/topic/detail/11094317
https://avg.163.com/topic/detail/11094285
https://avg.163.com/topic/detail/11094276
https://avg.163.com/topic/detail/11094258
https://avg.163.com/topic/detail/11094245
https://avg.163.com/topic/detail/11094216
https://avg.163.com/topic/detail/11094203
https://avg.163.com/topic/detail/11094200
https://avg.163.com/topic/detail/11094179
https://avg.163.com/topic/detail/11094158
https://avg.163.com/topic/detail/11094156
https://avg.163.com/topic/detail/11094157
https://avg.163.com/topic/detail/11094139
https://avg.163.com/topic/detail/11094130
https://avg.163.com/topic/detail/11094126
https://avg.163.com/topic/detail/11094097
https://avg.163.com/topic/detail/11094092
https://avg.163.com/topic/detail/11094085
https://avg.163.com/topic/detail/11094036
https://avg.163.com/topic/detail/11094039
https://avg.163.com/topic/detail/11094037
https://avg.163.com/topic/detail/11094025
https://avg.163.com/topic/detail/11094020
https://avg.163.com/topic/detail/11094017
https://avg.163.com/topic/detail/11093417
https://avg.163.com/topic/detail/11093973
https://avg.163.com/topic/detail/11093964
https://avg.163.com/topic/detail/11093939
https://avg.163.com/topic/detail/11093931
https://avg.163.com/topic/detail/11093911
https://avg.163.com/topic/detail/11093889
https://avg.163.com/topic/detail/11093882
https://avg.163.com/topic/detail/11093865
https://avg.163.com/topic/detail/11093835
https://avg.163.com/topic/detail/11093834
https://avg.163.com/topic/detail/11093811
https://avg.163.com/topic/detail/11093802
https://avg.163.com/topic/detail/11093789
https://avg.163.com/topic/detail/11093784
https://avg.163.com/topic/detail/11093776
https://avg.163.com/topic/detail/11093749
https://avg.163.com/topic/detail/11093711
https://avg.163.com/topic/detail/11093710
https://avg.163.com/topic/detail/11093698
https://avg.163.com/topic/detail/11093686
https://avg.163.com/topic/detail/11093682
https://avg.163.com/topic/detail/11093678
https://avg.163.com/topic/detail/11093673
https://avg.163.com/topic/detail/11093665
https://avg.163.com/topic/detail/11093607
https://avg.163.com/topic/detail/11093195
https://avg.163.com/topic/detail/11093188
https://avg.163.com/topic/detail/11093605
https://avg.163.com/topic/detail/11093603
https://avg.163.com/topic/detail/11093600
https://avg.163.com/topic/detail/11093596
https://avg.163.com/topic/detail/11093568
https://avg.163.com/topic/detail/11093565
https://avg.163.com/topic/detail/11093542
https://avg.163.com/topic/detail/11093537
https://avg.163.com/topic/detail/11093529
https://avg.163.com/topic/detail/11093496
https://avg.163.com/topic/detail/11092840
https://avg.163.com/topic/detail/11093466
https://avg.163.com/topic/detail/11093454
https://avg.163.com/topic/detail/11093446
https://avg.163.com/topic/detail/11093417
https://avg.163.com/topic/detail/11093407
https://avg.163.com/topic/detail/11093405
https://avg.163.com/topic/detail/11093383
https://avg.163.com/topic/detail/11093358
https://avg.163.com/topic/detail/11093349
https://avg.163.com/topic/detail/11093344
https://avg.163.com/topic/detail/11093326
https://avg.163.com/topic/detail/11093320
https://avg.163.com/topic/detail/11093295
https://avg.163.com/topic/detail/11093285
https://avg.163.com/topic/detail/11093260
https://avg.163.com/topic/detail/11093221
https://avg.163.com/topic/detail/11093195
https://avg.163.com/topic/detail/11093187
https://avg.163.com/topic/detail/11093188
https://avg.163.com/topic/detail/11093171
https://avg.163.com/topic/detail/11093167
https://avg.163.com/topic/detail/11092543
https://avg.163.com/topic/detail/11093117
https://avg.163.com/topic/detail/11093098
https://avg.163.com/topic/detail/11093067
https://avg.163.com/topic/detail/11093061
https://avg.163.com/topic/detail/11093058
https://avg.163.com/topic/detail/11093056
https://avg.163.com/topic/detail/11093050
https://avg.163.com/topic/detail/11093032
https://avg.163.com/topic/detail/11092914
https://avg.163.com/topic/detail/11093017
https://avg.163.com/topic/detail/11093015
https://avg.163.com/topic/detail/11092255
https://avg.163.com/topic/detail/11092962
https://avg.163.com/topic/detail/11092956
https://avg.163.com/topic/detail/11092941
https://avg.163.com/topic/detail/11092930
https://avg.163.com/topic/detail/11092914
https://avg.163.com/topic/detail/11092859
https://avg.163.com/topic/detail/11092858
https://avg.163.com/topic/detail/11092840
https://avg.163.com/topic/detail/11092834
https://avg.163.com/topic/detail/11092812
https://avg.163.com/topic/detail/11092806
https://avg.163.com/topic/detail/11092805
https://avg.163.com/topic/detail/11092804
https://avg.163.com/topic/detail/11092803
https://avg.163.com/topic/detail/11092802
https://avg.163.com/topic/detail/11092801
https://avg.163.com/topic/detail/11092786
https://avg.163.com/topic/detail/11092755
https://avg.163.com/topic/detail/11092751
https://avg.163.com/topic/detail/11092729
https://avg.163.com/topic/detail/11092722
https://avg.163.com/topic/detail/11092710
https://avg.163.com/topic/detail/11092708
https://avg.163.com/topic/detail/11092693
https://avg.163.com/topic/detail/11092686
https://avg.163.com/topic/detail/11092669
https://avg.163.com/topic/detail/11092661
https://avg.163.com/topic/detail/11092629
https://avg.163.com/topic/detail/11092623
https://avg.163.com/topic/detail/11092590
https://avg.163.com/topic/detail/11092582
https://avg.163.com/topic/detail/11092567
https://avg.163.com/topic/detail/11092566
https://avg.163.com/topic/detail/11092543
https://avg.163.com/topic/detail/11092531
https://avg.163.com/topic/detail/11092492
https://avg.163.com/topic/detail/11092480
https://avg.163.com/topic/detail/11092468
https://avg.163.com/topic/detail/11092469
https://avg.163.com/topic/detail/11092456
https://avg.163.com/topic/detail/11092454
https://avg.163.com/topic/detail/11092451
https://avg.163.com/topic/detail/11092444
https://avg.163.com/topic/detail/11092435
https://avg.163.com/topic/detail/11092434
https://avg.163.com/topic/detail/11092420
https://avg.163.com/topic/detail/11092400
https://avg.163.com/topic/detail/11092387
https://avg.163.com/topic/detail/11092379
https://avg.163.com/topic/detail/11092374
https://avg.163.com/topic/detail/11092358
https://avg.163.com/topic/detail/11092339
https://avg.163.com/topic/detail/11092334
https://avg.163.com/topic/detail/11092299
https://avg.163.com/topic/detail/11092282
https://avg.163.com/topic/detail/11092281
https://avg.163.com/topic/detail/11092266
https://avg.163.com/topic/detail/11092265
https://avg.163.com/topic/detail/11092264
https://avg.163.com/topic/detail/11092263
https://avg.163.com/topic/detail/11092260
https://avg.163.com/topic/detail/11092259
https://avg.163.com/topic/detail/11092258
https://avg.163.com/topic/detail/11092257
https://avg.163.com/topic/detail/11092256
https://avg.163.com/topic/detail/11092255
https://avg.163.com/topic/detail/11092254
https://avg.163.com/topic/detail/11092253
https://avg.163.com/topic/detail/11092252
https://avg.163.com/topic/detail/11092251
https://avg.163.com/topic/detail/11092249
https://avg.163.com/topic/detail/11092248
https://avg.163.com/topic/detail/11092247
https://avg.163.com/topic/detail/11092246
https://avg.163.com/topic/detail/11092245
https://avg.163.com/topic/detail/11092244
https://avg.163.com/topic/detail/11092243
https://avg.163.com/topic/detail/11092242
https://avg.163.com/topic/detail/11092241
https://avg.163.com/topic/detail/11092240
https://avg.163.com/topic/detail/11092239
https://avg.163.com/topic/detail/11092238
https://avg.163.com/topic/detail/11092237
https://avg.163.com/topic/detail/11092236
https://avg.163.com/topic/detail/11092235
https://avg.163.com/topic/detail/11092234
https://avg.163.com/topic/detail/11092233
https://avg.163.com/topic/detail/11092232
https://avg.163.com/topic/detail/11092231
https://avg.163.com/topic/detail/11092229
https://avg.163.com/topic/detail/11092227
https://avg.163.com/topic/detail/11092226
https://avg.163.com/topic/detail/11092224
https://avg.163.com/topic/detail/11092221
https://avg.163.com/topic/detail/11092219
https://avg.163.com/topic/detail/11092218
https://avg.163.com/topic/detail/11092217
https://avg.163.com/topic/detail/11092211
https://avg.163.com/topic/detail/11092207
https://avg.163.com/topic/detail/11092203
https://avg.163.com/topic/detail/11092205
https://avg.163.com/topic/detail/11092202
https://avg.163.com/topic/detail/11092195
https://avg.163.com/topic/detail/11092190
https://avg.163.com/topic/detail/11092187
https://avg.163.com/topic/detail/11092183
https://avg.163.com/topic/detail/11092179
https://avg.163.com/topic/detail/11092163
https://avg.163.com/topic/detail/11092162
https://avg.163.com/topic/detail/11092159
https://avg.163.com/topic/detail/11092156
https://avg.163.com/topic/detail/11092151
https://avg.163.com/topic/detail/11092142
https://avg.163.com/topic/detail/11092138
https://avg.163.com/topic/detail/11092136
https://avg.163.com/topic/detail/11092131
https://avg.163.com/topic/detail/11092129
https://avg.163.com/topic/detail/11092127
https://avg.163.com/topic/detail/11092113
https://avg.163.com/topic/detail/11092110
https://avg.163.com/topic/detail/11092107
https://avg.163.com/topic/detail/11092099
https://avg.163.com/topic/detail/11092104
https://avg.163.com/topic/detail/11092105
https://avg.163.com/topic/detail/11092103
https://avg.163.com/topic/detail/11092097
https://avg.163.com/topic/detail/11092093
https://avg.163.com/topic/detail/11092090
https://avg.163.com/topic/detail/11091068
https://avg.163.com/topic/detail/11092072
https://avg.163.com/topic/detail/11092067
https://avg.163.com/topic/detail/11092061
https://avg.163.com/topic/detail/11092058
https://avg.163.com/topic/detail/11092055
https://avg.163.com/topic/detail/11092041
https://avg.163.com/topic/detail/11092036
https://avg.163.com/topic/detail/11092027
https://avg.163.com/topic/detail/11092026
https://avg.163.com/topic/detail/11092010
https://avg.163.com/topic/detail/11092006
https://avg.163.com/topic/detail/11092002
https://avg.163.com/topic/detail/11091997
https://avg.163.com/topic/detail/11091994
https://avg.163.com/topic/detail/11091960
https://avg.163.com/topic/detail/11091936
https://avg.163.com/topic/detail/11091935
https://avg.163.com/topic/detail/11091923
https://avg.163.com/topic/detail/11091902
https://avg.163.com/topic/detail/11091889
https://avg.163.com/topic/detail/11091606
https://avg.163.com/topic/detail/11091837
https://avg.163.com/topic/detail/11091836
https://avg.163.com/topic/detail/11091830
https://avg.163.com/topic/detail/11091811
https://avg.163.com/topic/detail/11091800
https://avg.163.com/topic/detail/11091797
https://avg.163.com/topic/detail/11091784
https://avg.163.com/topic/detail/11091780
https://avg.163.com/topic/detail/11091738
https://avg.163.com/topic/detail/11091732
https://avg.163.com/topic/detail/11091680
https://avg.163.com/topic/detail/11091686
https://avg.163.com/topic/detail/11091682
https://avg.163.com/topic/detail/11091680
https://avg.163.com/topic/detail/11091651
https://avg.163.com/topic/detail/11091643
https://avg.163.com/topic/detail/11091622
https://avg.163.com/topic/detail/11091610
https://avg.163.com/topic/detail/11091606
https://avg.163.com/topic/detail/11091586
https://avg.163.com/topic/detail/11091581
https://avg.163.com/topic/detail/11091563
https://avg.163.com/topic/detail/11091549
https://avg.163.com/topic/detail/11091517
https://avg.163.com/topic/detail/11091515
https://avg.163.com/topic/detail/11091489
https://avg.163.com/topic/detail/11091478
https://avg.163.com/topic/detail/11091473
https://avg.163.com/topic/detail/11091443
https://avg.163.com/topic/detail/11091435
https://avg.163.com/topic/detail/11091411
https://avg.163.com/topic/detail/11091406
https://avg.163.com/topic/detail/11091383
https://avg.163.com/topic/detail/11091354
https://avg.163.com/topic/detail/11091351
https://avg.163.com/topic/detail/11091356
https://avg.163.com/topic/detail/11091353
https://avg.163.com/topic/detail/11091342
https://avg.163.com/topic/detail/11091335
https://avg.163.com/topic/detail/11091297
https://avg.163.com/topic/detail/11091288
https://avg.163.com/topic/detail/11091274
https://avg.163.com/topic/detail/11091272
https://avg.163.com/topic/detail/11091255
https://avg.163.com/topic/detail/11091237
