Self-managed and automatically reconfigurable stream processing

on YouTube ↗38m

" summary

Presents two implemented systems: SnailTrail, an online critical‑path profiler that reconstructs execution graphs from instrumented events and computes a weighted CP metric over sliding windows to rank bottlenecks, and DS2, a Rust controller integrated with Flink/Timely that predicts per-operator parallelism and reconfigures via Flink savepoints. Evaluation shows DS2 reaches correct configurations in one or two steps (reacting within seconds) versus six steps for a heuristic baseline, and both projects are open source on GitHub.

AI-written summary · about

" chapters (6)

1. 0:00Introduction to stream processing
2. 3:18Performance analysis of distributed dataflows
3. 16:39Snail Trail in action
4. 23:47Automated scaling decisions
5. 32:56DS2 scaling controller demo
6. 35:29Conclusion and future work

" related talks

• 21mSnailTrail: Generalizing Critical Paths for Online Analysis of Distributed DataflowsAndrea Lattuada, Sebastian Wicki, Moritz Hoffmann, Vasiliki Kalavri, John Liagouris, Timothy Roscoe, Desislava Dimitrova, Zaheer Chothia · NSDI '18
• 26mHow to Pick Your Next Streaming ArchitectureVladimir Bacvanski · Scale By the Bay 2017
• 38mFaster and Furiouser: Flink DriftTed Dunning · Flink Forward Berlin 2016
• 38mStreaming Fast and SlowCaito Scherr · Data Con LA 2020
• 19mSponge: Fast Reactive Scaling for Stream Processing with Serverless FrameworksSameh Elnikety, Won Wook Song, Byung-Gon Chun, Myeongjae Jeon, Taegeon Um · USENIX ATC '23
• 34mA journey of scaling high-volume low-latency stateful processingArti Pande · Flink Forward Global Virtual 2020
• 37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?Stephan Ewen · Flink Forward Global Virtual 2020
• 01h 00mDataflow Engines for Executing Cloud Applications: a Maslow's Hammer?Guido Salvaneschi, Asterios Katsifodimos, Marios Fragkoulis, Kyriakos Psarakis, Pedro Silvestre, Wouter Zorgdrager, Jim Verhijde, Adil Akhter · REBLS 2022