Big Data and Ruby
summary
The talk explains Big Data challenges for web engineers and compares Big Data frameworks, focusing on data locality, dataflow architecture, and the lambda architecture (and ways to simplify it). It discusses specific technologies—Hadoop, Storm, Summingbird, Spark—and how to use some from Ruby, describes experiments with a reactive computation abstraction at Oh My Stats, and explains why they implemented their own pure-Ruby recalculation framework (Flow.rb) and what goals and results they achieved.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:05What are big data
- 3:36Hadoop components
- 6:34MapReduce in Ruby with Hadoop Streaming
- 9:25Report example with Hadoop
- 13:39Hadoop pros and cons
- 16:14Streaming with Apache Storm
- 18:21Word count in Storm example
- 21:09Lambda architecture basics
- 24:20Lambda architecture in practice
- 27:57Flow and reactive computing
- 31:52Modern big data tools
- 34:03Q&A
