It's (not) down in the noise.
summary
Argument: use the Mann‑Whitney U test to prove small kernel performance changes are real, not “down in the noise.” Kernel microbenchmarks show skewed distributions with a near‑minimum mode, long tails and many minor modes, so mode vs mean matters and Student’s t is unsafe. Example: changing HZ from 1000 to 100 reduces slow‑path hits and yields U ≈ 0.75, showing a real effect. Advice: reboot between runs, run many repeats, report counts and spread.
AI-written summary · about
chapters (15)
- 0:00Start
- 0:20Peter Chubb
- 4:10Measuring a Single Value
- 6:56Standard Error on the Mean
- 9:56Central Limit Theorem
- 10:21The Central Limit Theorem
- 10:48Kernel Benchmark
- 16:50Phase Noise
- 17:14Internals of Clock Get Time
- 19:01Sequence Lock
- 21:59The Comparison
- 26:11Keep Your Measurements Independent
- 29:31Micro Benchmarks
- 31:03Reboot between Tests
- 33:32Which Functions Are Most Important in the Kernel
related talks
16mUsing Change Point Detection to Fight Noisy Benchmark Results
16mTaming Noisy Benchmark Results Using Change Point Detection
38mKnow your data: The stats behind your alerts
32mStatistics for Engineers
32mThe Subtle Art of Lying with Statistics
43mThe Subtle Science of Misleading with Statistics
29mIntuitive A/B Test Evaluations for Coders
32mLearning Statistics Will Save Your Life