Data Analytics Service Company and Its Ruby Usage
summary
Treasure Data runs a one‑stop analytics platform using Ruby/JRuby for collectors, console and the scheduler while heavy processing runs on JVM tools like Presto and Hadoop. The system handles around 12000000 jobs per day, with ~200k Hadoop jobs/day and 50k Presto queries/day. Job scheduling uses an RDBMS‑backed queue (called Perfect Scheduler) to provide per‑account isolation, priority, graceful restarts and at‑least‑once semantics; Ruby is chosen for dynamic job construction, heavy test coverage (RSpec) and operational flexibility.
AI-written summary · about
chapters (10)
- 0:00Introduction and self-introduction
- 3:55Why Ruby for data analytics
- 8:28Treasure Data’s internal systems
- 10:38Perfect Scheduler overview
- 16:20Job types and characteristics
- 19:09Perfect Queue system design
- 24:46Ruby for complex job assembly
- 27:56Testing and Ruby improvements
- 30:45Future system improvements
- 31:28Q&A on scalability
related talks
45mRuby for the soul of Big Data nerds
29mBig Data Enterprisy Analytics App and Ruby
41mJRuby and Big Data
49mData Platform Architecture, Evolution, and Philosophy
34mBuilding Data Driven Products with Ruby
51mPresto Open Source SQL Engine for Massive Scale Data Lakehouses
36mData Management with Ruby
48mContinuous Optimization for distributed BigData analysis