Processing data: Ruby or SQL?
summary
Compares doing numeric work in Ruby versus pushing it into the database using TimeScaleDB and the TimeScaleDB toolkit, measuring real run times and where time is spent. Generates ~50k records with generate_series, uses array_agg and lag or the toolkit pipeline operators to compute volatility; SQL path costs ~54 ms end‑to‑end while Ruby fetch+process takes ~190 ms (fetch ~116 ms, Ruby processing ~74 ms). Shows a 400k‑row downsampling where SQL returns under a second and Ruby takes ~3 seconds, concluding IO/serialization is the usual bottleneck.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:06Ruby vs SQL speed
- 7:50Data processing benchmarks
- 12:27Volatility algorithm vs SQL
- 16:27Ruby vs SQL benchmarks
- 21:07Query planning costs
- 24:01Hardware and data access
- 26:06Data pipes trade-offs
- 27:52Live demo: downsampling
- 29:58I/O vs CPU bottlenecks
related talks
25mPowerful Data: Making Rails Apps Faster with Aggregation
43mReal World Ruby Performance at Scale
03mAlgorithms
41mUsing Databases to Pull Your Application Weight
37mCache is King: Get the Most Bang for Your Buck From Ruby
44mSimulating the World with Ruby
28mData Science in Ruby? Is it possible? Is it Fast? Should we use it?
31m"Optimizing SQL Queries"