Demanding the impossible rigorous database benchm
summary
Argues that common tooling produces misleading PostgreSQL results and shows concrete fixes: PGbench can be a bottleneck (PGbench bottleneck), BPFTrace exposes a 150µs IO slow path, and open vs closed workload models change latency behavior. Recommends replacing means with median and quantiles, using nonparametric tests like Mann–Whitney, and randomized cross-traffic to equalize noise. Gives ballpark sample counts—stable CPU tests need ~10 runs, noisy IO tests may need 240 to 640 runs to reach 95% confidence.
AI-written summary · about
chapters (8)
- 0:00Database benchmarking challenges
- 1:44Unreliable benchmarking results
- 8:10PostgreSQL configuration for benchmarks
- 15:26Statistics for database benchmarks
- 22:31Non-normal performance distributions
- 27:25Reducing test noise and randomization
- 28:52Benchmarking takeaways
- 29:38Q&A on benchmarking
related talks
36mAutomated Performance Testing of Postgres
43mPostgreSQL in the cloud: Theory and Practice
29mPerformance Archaeology - 20 years of improvements
45mTuning Hardware and the Operating System: Benchmarking on PostgreSQL
50mPerformance Archaeology
44mDemystifying Benchmarks: How to Use Them To Better Evaluate Databases
53mVisualizing Postgres I O Performance for Development
48mModern SQL features in PostgreSQL