From Batch to Realtime with Hadoop
summary
Hadoop proves useful for long term batch storage but not instantaneous aggregates, so the architecture pairs HDFS MapReduce with HBase and a stream layer to bridge the gap. Facebook-style pipeline uses scribe/flume for collection, a Puma-like second for one-second aggregation, and HBase counters to serve front-end queries, achieving 10,000 inserts/sec on a single server and latency reduced from ~15min to 10–30s. For true subsecond needs it adds Storm for sub-second streaming while accepting eventual accuracy tradeoffs.
AI-written summary · about
chapters (7)
- 0:00Introduction and speaker background
- 10:18Hadoop ecosystem overview
- 17:17Batch processing limitations
- 23:03HBase for real-time storage
- 32:37Stream processing with Storm
- 37:18Counter vs aggregate approaches
- 40:07Q&A session
related talks
36mReal Time Big Data Analytics with Kafka, Storm, HBase #bbuzz
53mBig Data: from mammoth to elephant
19mLog Events @ Twitter: Challenges of Handling Billions of Events per Minute
22mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors
08mReal-Time Data Systems: Speed, Scale, and Resilience
28mBest of Stream + Batch: When Streaming Analytics Isn’t
12mthe Rapidly Changing Hadoop Ecosystem
32mBig Data Processing : Where do we go next ?