Taking Hadoop Realtime with Summingbird
summary
Summingbird is an open source Scala library that runs the same analytics DAG on Hadoop (Scalding) and Storm and merges results with a batch+realtime merge so clients see mostly-Hadoop-backed live counts. It encodes aggregation types as a monoid abstraction, lets stores accept monoid-valued payloads, and uses approximate structures like a Count-Min sketch with a top-K heap to track embedded-tweet impressions. Deployed use includes advertising dashboards and a reported ~5x speed improvement, and the hybrid mode reliably bounds realtime error.
AI-written summary · about
chapters (11)
- 0:00Introduction to Summingbird
- 1:04Streaming mapreduce vision
- 3:28The problem of data scale
- 6:28Summingbird architecture
- 10:51Core abstractions
- 13:25Abstract algebra and monoids
- 16:50Compiler-assisted analytics
- 19:03Batch and real-time merging
- 22:59Practical case study
- 27:35Production application examples
- 30:13Future directions and Q&A
related talks
51mStream all the things! - life at the end of the firehose
01h 25mApache Spark for Big Data Processing
56mSpark-driven Audience counting
52m100% Big Data. 0% Hadoop. 0% Java.
37mBig Data Analytics, Machine Learning, Real Time Processing
01h 35mDeep Dive into the Big Data Landscape
30mSmart Streams: A Real-time framework for scoring Big and Fast Data
44mData Science at Scale with Apache Spark