Google Napa: Scalable Data Warehousing with Robust Query Performance
summary
Google Napa is a Google internal data warehouse that ingests trillions of rows per day and serves billions of queries with sub‑second median latency by betting on materialized views (hundreds-to-thousands, all strongly consistent) and decoupling ingest, storage/indexing and query serving. A queryable timestamp bounds freshness and guarantees a fixed, small number of deltas read (an 8‑view experiment covers a real workload), F1 Query and Colossus remain the storage/compute substrate, views are incrementally maintained, and a controller automates SLAs and tail mitigation so dashboards stay up with bounded staleness.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 5:16Robust query processing
- 12:22Why is this hard
- 13:43Why is Napa different
- 17:35Does Napa model updates
- 23:10Resolving conflicting goals
- 24:22How Napa works
- 25:37Variable timestamps
- 28:25Pushpull
- 29:52Experimental Framework
- 31:02F1 Query
- 33:40Challenges
- 34:57Query serving
- 38:47Middle Layer
- 40:47Views
- 41:48View Order
- 42:16Merge Base
- 43:08Tradeoffs
- 46:33Decoupling
- 48:25Summary
- 49:33QA
- 54:08Complex Views
- 55:58Distributed Cache vs Local Cache
- 57:48Cost
- 59:23Incremental maintenance
related talks
30mData Warehouse in Google Cloud
05mThe Future of Data Warehousing
52mBuilding the World's Largest Enterprise Data Warehouse with BigQuery
28mThe architecture behind modern cloud data warehouses
55mFollowing Google or Don't Follow the Followers, Follow the Leader
28mBuilding A Governed Data Lake
32mHow open source is re-shaping the cloud data warehouse landscape
41mOn Data: Sky above, sand below, peace within