Meces: Latency-efficient Rescaling via Prioritized State Migration for Stateful...
summary
Meces implements fetch-on-demand and prioritized hot-key migration on Apache Flink to rescale stateful stream jobs with minimal latency. It migrates state at record granularity, shards key groups into subgroups, and uses gradual micro-batch migration plus a controller message protocol for exact consistency. Evaluations on benchmarks show latency peaks under 1,000 ms and often below 400 ms versus 8,000–10,000 ms for prior methods, with negligible steady-state overhead and no disruption to non-rescaling periods.
AI-written summary · about
chapters (7)
- 0:00Start
- 0:38Motivation
- 1:41Related Work
- 3:22Prioritized Migration
- 5:37Meces: Design and Mechanisms
- 12:26Evaluation
- 15:48Conclusion
related talks
34mA journey of scaling high-volume low-latency stateful processing
38mAutoscaling Flink at Netflix
09mLoad Imbalance
15mStreaming, Fast and Slow: Mitigating Watermark Skew in Large, Stateful Jobs
43mDynamic Scaling - How Apache Flink adapts to changing workloads
18mTowards More Efficient and Adaptive Scheduling for Flink Batch
31mTale of Stateful Stream to Stream Processing
19mSponge: Fast Reactive Scaling for Stream Processing with Serverless Frameworks