On Benchmarking With Ruby
summary
Benchmarks claim stable numbers by measuring repeatedly, using Perfer to store runs and compare commits, and choosing clocks carefully — prefer process CPU time or monotonic clocks over wall time. MRI 2.1 shows a generational GC speedup that cuts GC overhead dramatically. Use iterative runs, report the median with median absolute deviation for error, and require minimum run lengths; an example result is about 140000 iterations/sec with ~2.5% error.
AI-written summary · about
chapters (25)
- 0:00Start
- 1:18Motivation
- 2:26Factors
- 2:53JIT: JRuby
- 4:17Memory, GC and loading code
- 5:05GC in 2.1
- 6:10Context
- 8:41What time, which clock?
- 14:22Statistics
- 15:04Distribution and outliers
- 15:34Central tendency: mean, median
- 16:10Measures of dispersion
- 17:19The standard library
- 17:59MRI benchmark/driver rb and the suite
- 19:18Perfer vocabulary
- 20:12Measurement API iterations
- 20:39Measurement API input size
- 21:09Measurement CLI
- 21:50Measurement: metadata recorded
- 22:40Reporting
- 22:50Comparison
- 23:21Visualization
- 24:03Current status
- 25:17Big time
- 26:30Questions
related talks
43mBenchmarking Ruby
29mPractical guide to benchmarking your optimizations
18mPerformance Optimization in Ruby
11mRuby Racing: Challenging Ruby Methods
53mWriting Fast Ruby
29mHow to Performance
02h 44mMeasure, don’t guess: benchmarking stories from the trenches
42mTweaking Ruby GC Parameters for Fun, Speed, and Profit