Disco: Efficient Distributed Window Aggregation
summary
Disco implements distributed window aggregation by slicing streams, pre-aggregating at leaf and intermediate nodes, and merging intermediate results at the root; evaluation on tumbling 1s windows (120s runs) shows throughput scales linearly with node count and a single child processes ~1M events/sec, already outperforms centralized in two-node setup. Supports combinable aggregates (sum/avg) via partial sums and counts, and handles holistic functions (median) by sending each slice once. Network bytes drop sharply versus central ingestion; code and paper are on Github and EDBT 2020.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 0:37Current State
- 1:59Our Solution
- 3:06Stream Slicing
- 5:47Distributed Aggregation
- 9:11Example
- 14:47Evaluation
related talks
24mScotty: Efficient Window Aggregation with General Stream Slicing
15mProduction Multi-Party Computation via the Distributed Aggregation Protocol
35mScotty: Efficient Window Aggregation for your Stream Processing System
18mIntroducing WinBro for Scalable Streaming Graph Partitioning
34mGoing deep with Spark Streaming #bbuzz
22mJoining Dozens of Data Streams in Distributed Stream Processing Systems
52mThe Computer Science behind a modern distributed data store
40mCentralized Counters in a Distributed World Event Based Realtime Statistics using eTally