Small Data: Databases in the real world
summary
Advice: target teams of 1–100 engineers, prioritize migrations and deployment speed over premature scaling, prefer Postgres preferred for lockless schema changes, use add nullable column for free then backfill, run databases on dedicated beefy boxes not noisy VMs, add read replicas and pin writes to master to avoid stale reads, start sharding with 2048 logical shards and consistent hashing, denormalize via nullable backfills, and accept some inconsistency because replication lag >10s breaks things rather than losing availability.
AI-written summary · about
chapters (18)
- 0:00SMALL DATA DATABASES IN THE REAL WORLD
- 2:00Scalable designs are a tradeoff: NOW vs LATER
- 3:02Small company? Agency? Focus on ease of change, not scalability
- 3:32Rapid development Continuous deployment Hardware choice Scaling 'breakpoints
- 3:52Rapid development It's all about schema change overhead
- 6:45Continuous deployment It's 11pm. Do you know where your locks are?
- 7:251-to-1 relation and backfill
- 9:49Hardware choice ZOMG RUN IT ON THE CLOUD
- 10:52Memory is king Your database loves it. Don't let other apps steal it.
- 13:24Sharding point Vertical split Consistency leeway
- 14:37Migration plan Implement consistent hashing on primary key Make large number of logical shards (2048?) Map logical shards to single physical shard Migrate shards using replication
- 16:29Migration plan Replicate database to new server Route split tables there, disable replication or - Slowly backfill new datastore with fallback lookup
- 18:16Migration plan Add NULL fields to dependent tables App code to fetch and fill if not present Possibly prefill on save of new items
- 19:34Consistency leeway Can you take inconsistent views?
- 20:40Migration plan Change your site! Talk to your designers! Deliberately introduce inconsistency!
- 22:21Big Data isn't one thing It depends on type, size, complexity. throughput, latency...
- 23:14Focus on the current problems Future problems don't matter if you never get there
- 24:27Good architecture affects product You're not writing a system in a vacuum
related talks
24mDatabases beyond the tutorials
35m10 things a developer should know about databases
31mHelp! My Database Is The Bottleneck
48mWeb scale for the rest of us
36mPostgres at any Scale
28mTobias Petry :: Scaling Databases
46mThe Lost Art of Database Design
54mLessons learned from refactoring multi terabyte databases for the last 10 years