"Distributed stream processing in Ruby"

on YouTube ↗42m

" summary

Shows how Apache Storm topologies are built from Ruby spouts and bolts, using a community Ruby gem, Kafka spouts and HTTP calls to back-end services. Presents two real pipelines: an interaction counter that sustains ~100k requests per minute with end-to-end persist under 1 ms, and an email-scoring pipeline that runs a ML model and issues alerts. Claims Storm can benchmark million tuples per second, cluster of 2 Nimbus 2 ZK 4 supervisors runs 25 topologies for <$1,000/month.

AI-written summary · about

" chapters (12)

1. 0:00Introduction to stream processing
2. 2:58Batch vs stream processing
3. 7:19Stream basics and terminology
4. 15:27Processing streams in three ways
5. 19:04Stream processing use cases
6. 23:28Apache Storm architecture
7. 29:24Storm in Ruby overview
8. 30:55Real-world example tracking
9. 37:03Email sentiment analysis pipeline
10. 39:20Costs and infrastructure
11. 40:02Alternatives to Apache Storm
12. 41:07Q&A

" related talks

• 37mStorm, a Distributed Real-time Data Processing SystemMariusz Gil · Confitura 2013
• 45mRuby for the soul of Big Data nerdsAbhishek Parolkar · RubyConf India 2012
• 35mReal Time Marketing with Kafka, Storm, Cassandra & Spark #bbuzzVolker Janz · Berlin Buzzwords 2016
• 36mReal Time Big Data Analytics with Kafka, Storm, HBase #bbuzzAmeya Kanitkar · Berlin Buzzwords 2015
• 31mHelp I need a stream processor - learning how to chose ...Andrew Psaltis · Berlin Buzzwords 2016
• 26mBuilding Stream Processing Applications with Ruby & MeroxaAli Hamidi · RubyConf 2022
• 30mFive Challenges in Real-time Stream Processing and Five SolutionsFawaz Ghali · SREday 2023 London
• 41mThe Engineer's Guide to StreamingIlya Ganelin · Data By the Bay 2016