FalconFS: Distributed File System for Large-Scale Deep Learning Pipeline
summary
FalconFS replaces client-side metadata caching with a stateless-client architecture that sends full paths to servers and resolves them using hybrid metadata indexing plus lazy namespace replication, concurrent request merging, and a VFS shortcut for kernel compatibility. Benchmarks vs CephFS and Lustre report up to 5.72× throughput for small file IO and 12.81× for model training; it runs in production on 10,000 NPUs for a year and claims >50% storage cost reduction.
AI-written summary · about
chapters (8)
- 0:00FalconFS introduction
- 2:24Metadata bottleneck
- 4:33Stateless client path resolution
- 7:45FalconFS design
- 10:33Namespace replication strategy
- 11:36WEFS shortcutting
- 12:18Performance evaluation
- 13:46Conclusion
related talks
31mScaling Autonomous-Driving R&D: A 100 PB, 100 Billion-File Open-Source Elastic File...
29mCubeFS in Action: Empowering Users Through Case Studies
33mThe Evolution and Innovation of CubeFS
19mFlatFS: Flatten Hierarchical File System Namespace on Non-volatile Memories
36mOptimizing Ceph Object Storage for Analytics and AL/ML Workloads - Overview and Future Plans
37mAn Initial Look at Deep Learning IO Performance
50mPerformance comparison of distributed file systems
59mSimplifying Big Data Management with Apache Falcon