Inside S3
summary
Description of Yandex’s S3-compatible storage that holds billions of files, focusing on the challenges of managing very large volumes of data and metadata. Explains using many PostgreSQL shards to store metadata and discusses the storage architecture and operational lessons from that deployment.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:13What is S3P and Goose
- 2:38S3 requirements and scale
- 5:06MDS storage overview
- 6:51Sharding and replication
- 8:15Dynamic sharding approach
- 10:43Mover process
- 13:35Chunk size and splitter
- 15:44Database types in S3P
- 17:52Object counting and billing
- 20:54Sequential key pattern issue
- 23:30Khunks for load balancing
- 26:22Khunks impact on listings
- 31:20Q&A
- 43:17Summary
related talks
54mOK S3: Building the System Ourselves
01h 01mOK S3
39mMailing Adventures with PostgreSQL: Taming 650+ Shards and Surviving
40mDatabases in the Cloud
50mSQL 2022 and S3 – Dell’s overview of consuming object data via Microsoft SQL
50mA Homegrown S3 Server: Challenges of Building Stable Storage
40mStorage Architecture for Yandex.Direct Advertising Objects
54mPrimary storage for SaaS solutions