The Design of a Modern Distributed Object Storage
summary
A description of how to move from vertical to horizontal scaling for large amounts of static user content by adopting distributed Object Storage with an S3/REST interface. It explains common architecture components—metadata servers (and their redundancy), request routers, and storage nodes with local disks—and describes the criteria used to evaluate options, leading to choosing and deploying LeoFS in production to store several terabytes.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 4:22Technology selection challenges
- 10:12Caching in Japanese tech
- 18:30Distributed storage in action
- 25:50Developer collaboration and latency
- 28:00Fault tolerance and memory
- 30:15File consumers overview
related talks
47mA Distributed, Scalable and Highly Reliable Data Storage System
50mA Homegrown S3 Server: Challenges of Building Stable Storage
32mElevating Scalable Object Storage: A Comprehensive Exploration of Ozone’s Trailblazing Capabilities
30mDistributed system design
33mPredicting the Future of Distributed Systems
21mNew Techniques to Improve Small I/O Workloads in Distributed File Systems
41mIntroduction to distributed systems
17mSingularFS: A Billion-Scale Distributed File System Using a Single Metadata Server