The Evolution of a Planetary-scale Distributed Database
summary
NRDB claims a global, cloud-agnostic system ingesting exabytes of data across tens of thousands of JVMs while keeping query median near 50ms response time. Early design relied on Kafka (at one point the largest Kafka broker cluster), later moved to Kubernetes, Cloudflare workers and a cell architecture. Storage shifts make S3 primary, cutting local disk to 5% of cost. Bin‑packing of big tenants, smart query routing and stateless query cells are built and operate in production.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:34Outline
- 1:44Conceptual Architectural Design
- 2:47Product Design Goals
- 6:07In the beginning
- 8:21Decoupling Ingest & Query, v1
- 9:38Ingest & Query Routing, v1
- 16:11Cell Architecture & Cloud Migration
- 19:53Ingest & Query Routing, v2
- 25:11Problem: Bin-packing is (NP-)Hard
- 26:05Decoupling Storage
- 27:25Decoupling Ingest & Query, v2
- 28:50Regions & Cloud Providers
- 32:15Your customer doesn't care about your architecture
- 33:00Decoupling Ingest & Query, v3
- 33:45Decoupling Metadata
- 34:38Differentiated Query Cells
related talks
11mDatabase Evolution: Partitioning and Indexing
57mMulti Tenant NoSQL and NewSQL Cloud data Design Patterns Deep dive
39mRethinking RDBMS Data Migration
53mFrom RDBMS to NoSQL (PRT314)
29mModern Database Architecture: From SQL to NoSQL & Distributed SQL
28mNSDB (Natural Series DB)
18mBuilding a Scalable, Resilient System for Real-Time Fraud Detection
26mCloud Native Database Meetup Japan: The State of Distributed Database in Japan