mruby gateway for huge amount of realtime data processing
summary
TreasureData replaces a Rails+Fluentd front end with H2O + mruby gateway and a custom aggregator to handle rapidly growing realtime imports, deploying on AWS auto scaling. Patches add mruby-friendly parallel and asynchronous processing, implemented with mruby async via fibers and tasks plus channel-like primitives so uploads can continue in background. Notes cover mruby C-API pitfalls (string copying and GC) and show microbenchmarks of the new gateway. Patched H2O for parallel mruby handlers
AI-written summary · about
chapters (15)
- 0:00Introduction
- 2:16Treasure Data overview
- 6:03Streaming data import challenges
- 9:22New system design
- 12:20Gateway server role
- 15:57Why mruby and H2O extension
- 17:29Basic mruby extension usage
- 19:24Async HTTP request implementation
- 23:21mruby and C extension integration
- 26:51Async function implementation
- 30:24Parallel HTTP request challenges
- 33:43Callback channel idea
- 36:40Task function implementation
- 41:28mruby extension pitfalls
- 44:12References and summary
related talks
45mRuby for the soul of Big Data nerds
30mHTTP Programming with mruby
41mJRuby and Big Data
34mBuilding Data Driven Products with Ruby
26mBuilding Stream Processing Applications with Ruby & Meroxa
34mData Analytics Service Company and Its Ruby Usage
25mRuby and PostgreSQL, a love story
05mRuby startup demo – Nagendra Dhanakeerthi