Enough statistics so that Zed won't yell at you
summary
Explains concrete tests and tools rather than theory giving z-test for proportions when comparing conversion rates and t-test and p-value for benchmarked latencies, shows httpperf benchmarks against mongrel and warns EC2 virtualization as a confounding variable, reports the practical rule of thumb of about 30 samples for means, recommends R for analysis and automated harnesses for reproducible runs, and demonstrates how to isolate single changes so measured performance differences actually mean something.
AI-written summary · about
chapters (15)
- 0:00Start
- 0:44Caveats
- 1:16Why Statistics Came About
- 2:19Standard Normal Distribution
- 6:09Sampling
- 9:03Z-Test
- 13:08Http Perf
- 17:17T-Test
- 20:10Confounding Variables
- 22:21Review
- 23:06Automate Your Test Harness
- 23:32Resources
- 24:00Benchmarking with Http Perf
- 27:42The Central Limit Theorem
- 29:28The Power Test
related talks
29mPractical guide to benchmarking your optimizations
43mReal World Ruby Performance at Scale
37mA/B testing at Zalando: concepts and tools
43mBenchmarking Ruby
39mPractical Unix for Ruby & Rails
48mThere is Such a Thing as Too Much Testing
41mRuby Monitoring State of the Union
33mEnsuring High Performance for Your Ruby App