Aggregator: MapReduce in the type system
summary
Claim: expressing the MapReduce reduction as a typed Aggregator separates what is computed (input A, accumulation B, output C) from how it runs. The talk demonstrates Aggregator trait in Algebird with compose prepare/present and associative reduce, swaps exact sets for fixed‑space sketches, and composes features. It shows production use at Stripe for merchant fraud features and a hybrid execution using Summingbird for hybrid batch/stream to run the same logic on Storm and Hadoop.
AI-written summary · about
chapters (30)
- 0:00Introduction
- 0:42Stripe Data Domain
- 1:48Data Analysis
- 3:19Rank Comments
- 4:07Grouping
- 4:54MapReduce
- 5:43MapReduce API
- 6:28MapReduce Example
- 7:19What is a MapReduce Platform
- 8:24MapReduce Phases
- 9:06Expressing MapReduce
- 10:05Aggregating MapReduce
- 12:35Aggregator
- 13:38Pseudo Scala
- 14:10Computation Type
- 14:48Scala Collections Example
- 15:29Aggregator Structure
- 19:39Leveraging the Standard Library
- 21:09ComposePrepare
- 23:32Examples
- 25:18Merchant Fraud
- 25:55Predictive Model
- 26:28Features
- 29:02Input Type
- 29:36Present Method
- 31:23Problem
- 32:02summingbird
- 32:51example
- 34:15recap
- 35:06summary
related talks
38mAdd ALL the things: abstract algebra meets analytics
39mUsing Monoids for Large Scale Business Stats
33mHow PostgreSQL aggregates work and how to create and use custom...
25mMergeable Summaries for High Scale, High Cardinality Analytics
43mBeyond Shuffling: Scaling Apache Spark
57mAlgebraic Databases
44mIntermediate Relational Algorithm Design
33mAggregating metrics in-flight: challenges and opportunities