Systems @Scale Summer Q&A Week 3
summary
Two storage stacks are compared by implementation and tradeoffs not marketing: one (RTD/arcdb) is rebuilt for NVMe and fast datacenter networks, uses a persistent page mapping table with coroutine-driven IO, accepts higher complexity and random-read amplification so it is not suited for HDD, and guarantees writes once the log ack returns; the other (Tectonic/Pangu) presents append-only design with single-writer semantics per file, disaggregated metadata layers that increase round trips, and a background rebalancer that moves chunks when nodes are added.
AI-written summary · about
chapters (20)
- 0:00Systems at Scale Q&A
- 1:11Live Q&A
- 2:17Key-value store in Tectonic/Pangu
- 5:10ArcDB for Pangu metadata
- 5:52Concurrency for file appends
- 6:56Heterogeneous storage in Pangu
- 8:23Metadata to user data ratio
- 9:25Facebook’s new file system
- 11:13Tectonic migration challenges
- 13:42Failover data loss thresholds
- 16:53KV store persistence guarantees
- 17:34Tectonic data center limits
- 20:02Pangu global storage approach
- 21:03ArcDB design trade-offs
- 23:33ArcDB correctness guarantees
- 25:15Tectonic metadata trade-offs
- 27:23Tectonic append-only design
- 29:51ArcDB vs RocksDB
- 32:46Cluster auto-scaling/repartitioning
- 34:58Closing remarks
related talks
29mUnification of Temporary Storage in the NodeKernel Architecture
42mPostgreSQL Distributed: Architectures & Best practices
52mThe Computer Science behind a modern distributed data store
19mBuilding a Performant Configuration-Store Layer
33mPredicting the Future of Distributed Systems
36mKeynote: State of the Cephalopod
36mStreamlio: Towards a Unified End-to-end Solution
21mBuilding Distributed Platforms at Extreme Scale on Fast Changing Data