Scotty: Efficient Window Aggregation for your Stream Processing System

on YouTube ↗35m

" summary

Scotty is a standalone window operator that uses stream slicing and partial pre-aggregates to avoid assigning each event to every overlapping window (example: a 10‑minute window sliding every second would otherwise create 600 buckets per event). It exploits function properties (invertible, associative), supports sliding, session, time/count measures, and handles out‑of‑order updates. Benchmarks show Scotty embedded in Flink/Storm/Beam reaches up to one order of magnitude speedups and sustained throughput near 2 million tuples/sec; code and connectors are on GitHub.

AI-written summary · about

" chapters (15)

1. 0:00Intro
2. 1:19Stream Processing Systems
3. 2:50Aggregations in Stream Processing Pipelines
4. 4:54Apache Flink - Stateful Stream Processing
5. 5:56Motivation
6. 7:14Research Background
7. 8:59Stream Slicing Example
8. 13:56Impact of Workload Characteristics
9. 17:37Key-Facts
10. 19:34Scotty Core
11. 21:00Benchmark
12. 22:08Using Scotty on Flink
13. 23:22Implement your own Aggregations Functions
14. 25:46Upcoming Research Projects
15. 28:29Scotty Window Processor

" related talks

• 24mScotty: Efficient Window Aggregation with General Stream SlicingPhilipp Grulich, Jonas Traub · Flink Forward Europe 2019
• 19mDisco: Efficient Distributed Window AggregationLawrence Benson · Flink Forward Global Virtual 2020
• 39mAdvanced Stream Processing on the EdgeEduardo Silva · linux.conf.au 2020
• 45mData Stream Processing Concepts and ImplementationsMatthias Niehoff · Devoxx Belgium 2017
• 31mHelp I need a stream processor - learning how to chose ...Andrew Psaltis · Berlin Buzzwords 2016
• 40mRobust Stream Processing with Apache FlinkJamie Grier · Flink Forward Berlin 2016
• 26mHow to Pick Your Next Streaming ArchitectureVladimir Bacvanski · Scale By the Bay 2017
• 38mSelf-managed and automatically reconfigurable stream processingVasiliki Kalavri · Flink Forward Europe 2019