Avoiding Bad Database Surprises: Simulation and Scalability
summary
Argues that prevention is possible by simulate early and simulate often, using Python to build a tiny processing nucleus, a JSON Schema–driven generator of realistic synthetic data, and real runs against a real DB instance. Shows tradeoffs: index changes, compression, and concurrency measured with tens of millions of rows (example: 50M) rather than guesses, and describes generating enums, date formats, credit‑card patterns, and spinning cloud servers to gather hard latency and throughput numbers.
AI-written summary · about
chapters (12)
- 0:00Database performance horror stories
- 4:04Databases as bottlenecks
- 7:23Simulating data early
- 10:36Performance testing with Python
- 13:09Costs of large data volumes
- 17:04JSON schema for modeling
- 22:03Synthetic data from JSON schema
- 27:05Realistic data generation
- 31:05Avoiding assumptions with measurement
- 32:10Technical spikes and engineering
- 34:01Cloud performance testing
- 35:04Q&A
related talks
29mHow to build a data pipeline without data
24mDatabases beyond the tutorials
53mMake Your Database Dream of Electric Sheep: Designing for Autonomous Operation
28mSmall Data: Databases in the real world
51mThe Lost Art of Database Design
41mUsing Databases to Pull Your Application Weight
02h 42mSQL (The Return): Demystifying Preconceptions and Using the DB Effectively
31mHelp! My Database Is The Bottleneck