Benchmarking: You're Doing It Wrong
summary
Benchmarking that starts measuring immediately on one image accessed 1,000 times, takes three runs and reports the mean is misleading: caches (L1/L2/L3, app cache, memcache, S3) and warm-up and JIT change steady state, periodic GC and power‑scaling distort samples, and coordinate emission hides missed requests. Use realistic workloads, force or exclude initial GC, collect many samples (often 40+ runs), report percentiles and histograms (not just mean or 50th/99th), watch for bimodal distributions and 1% outliers that at scale break pages with 100 images.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:19To Write Good Benchmarks...
- 2:23What's a Benchmark
- 4:13Website Serving Images
- 5:26It's Caches All The Way Down
- 7:12Caches in Benchmarks
- 11:20Wrong About the Machine
- 19:16Multimodal Distribution
- 20:13Wrong About Stats
- 23:31Programmers waste enormous amounts of time thinking about ... the speed of noncritical parts of their programs... Forget about small efficiencies ...97% of the time
- 23:57Wrong About What Matters
- 26:32Becoming Less Wrong
- 29:47Microbenchmarking: Blessing & Curse
- 31:47Non-Constant Work Per Iteration
- 33:49Outliers
- 34:25Workload
related talks
02h 41mMicro Benchmarking - The Art of Realizing One is Wrong
02h 44mMeasure, don’t guess: benchmarking stories from the trenches
38mYour benchmark may not guide a real application performance
21mBenchmarking Like a Boss
27mOn Benchmarking With Ruby
43mStop Guessing and Start Measuring - Benchmarking in Practice
06mIGNITE: How to benchmark … poorly
40mBenchmarking: Repeatable & Comparable