Building An Elastic Query Engine on Disaggregated Storage
summary
Snowflake decouples compute from persistent storage and runs analysis on 70 million queries over 14 days showing intermediate-data sizes span several orders of magnitude and are poorly correlated with CPU or input size. Ephemeral storage is co‑located with VMs using DRAM+SSD+spill to S3, caches persistent files with write‑through and consistent hashing, and avoids reshuffles with lazy consistent hashing that fetches from S3 on first access. About 20% of warehouses resize in production, sometimes up to 100×, and per‑second billing breaks the economics of prewarmed VM pools.
AI-written summary · about
chapters (14)
- 0:00Traditional Shared-Nothing Architectures
- 1:57Query Engines on Disageregated Storage
- 3:15Diversity of Queries
- 4:04Query distribution over time
- 4:28High-level architecture
- 6:07Ephemeral Storage System
- 7:28Intermediate Data Characteristics
- 9:05Persistent Data Caching
- 11:42Lazy Consistent Hashing
- 13:16Do customers exploit elasticity in the wild?
- 14:41Resource Utilization
- 15:46Alternate design: Resource Sharing
- 17:27Resource sharing challenges
- 18:53Conclusion
related talks
13mLightning Talk: Highlights of the Snowflake Elastic Data Warehouse
41mThe New Generation Of Data Stores
20mSnowflake concepts & architecture
33mNext Gen of Datastores: Hot Queries, Cold Storage
52mData storage for modern applications - Mike Lehan
21mCaching Snowflake data with Aerospike for advanced analytics
28mBuilding A Governed Data Lake
28mThe architecture behind modern cloud data warehouses