Norikra: SQL Stream Processing in Ruby
summary
Norikra is an open source SQL stream processor implemented in JRuby on JVM that runs as a server and accepts queries and UDFs at runtime. It aggregates live JSON event streams (counts per 5 seconds, sums, group by name), persists summaries to MySQL and pushes results to dashboards and email, and integrates with Hadoop/Hive and BigQuery. Used in LINE (≈130 million users) for error log summarization and hourly prompt reports, running in production.
AI-written summary · about
chapters (23)
- 0:00Norikra SQL stream processing
- 1:04Speaker background
- 2:28Data analytics challenges at LINE
- 5:49Fluentd limitations for streams
- 8:00Why Norikra for streams
- 8:22Norikra core features
- 9:25Installing Norikra
- 10:33Feeding events into Norikra
- 11:21SQL on event streams
- 12:32Schema changes in Norikra
- 13:37Aggregation in Norikra SQL
- 14:45Ordering query results
- 15:37Multiple schemas in Norikra
- 16:22Advanced Norikra SQL
- 17:29Error log summarization
- 19:42Norikra vs Hadoop
- 20:52Daily reports with Norikra
- 21:36Google engineer’s use case
- 22:59Lambda architecture with Norikra
- 24:08Esper and JRuby foundations
- 25:14Open source plugins
- 26:16JRuby powers Norikra
- 27:25Conclusion and try Norikra
related talks
45mRuby for the soul of Big Data nerds
30mRuby in the Billions
42m"Distributed stream processing in Ruby"
05mRuby startup demo – Nagendra Dhanakeerthi
26mBuilding Stream Processing Applications with Ruby & Meroxa
41mJRuby and Big Data
34mBuilding Data Driven Products with Ruby
22mJRuby With Java Code in Data Processing World