Add ALL the things: abstract algebra meets analytics
summary
Claim: many analytics problems reduce to a single abstraction so one aggregation engine handles counts, max, top‑k, histograms, uniques and frequencies. Tools and ideas: commutative monoids as the core, concrete pieces like statsd, Algebird/Scalding, HyperLogLog (~2% error with 1k counters), Count‑Min Sketch and Bloom filters, and transform‑into‑space tricks (sum+count for averages, mins for uniques, matrix+min for frequencies). Built examples include a Scala library (Algebird) with a small UNIXy shim used in production at Stripe and the techniques work efficiently in distributed realtime rollups.
AI-written summary · about
chapters (13)
- 0:00Intro
- 1:26Motivation
- 1:38Statsd
- 2:35Addifier
- 7:42Max
- 9:06Properties
- 12:36Example
- 14:51Averages
- 20:45Unique values
- 22:52Average distance
- 30:06Account minsketch
- 33:04Abelian groups
- 36:39References
related talks
39mUsing Monoids for Large Scale Business Stats
35mExpressing complex data aggregations with Histogrammar
25mMergeable Summaries for High Scale, High Cardinality Analytics
36mAggregator: MapReduce in the type system
32mKnow your data the stats behind the Alters
48mHyper Speed! When Big Data Blooms
33mRelevant Data Analysis: Apache Solr Analytics #bbuzz
41mBuilding an Open Source Streaming Analytics Stack with Kafka and Druid