Resource Efficient Stream Processing Platform with Latency-Aware Scheduling
summary
A platform prototype implemented on Spark Structured Streaming shares executors across multiple streaming apps and uses task-level scheduling with a black-box, latency-driven policy to reassign CPU dynamically. The system claims a 36% reduction in required CPU cores (39 → 25) versus separate clusters, evaluates with pseudo IoT services, and tests three schedulers including latency-aware EDF and a priority-weighted EDF. Under overload, the priority EDF cuts SLA violations by about 90% compared to FAIR.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:22Background
- 1:50Challenges
- 3:13Proposal: Multi-application platform
- 4:43Proposal: Architecture Overview
- 5:44Proposal: Latency-aware task scheduler
- 10:13Evaluation
- 14:53Conclusion
related talks
19mStarburst: A Cost-aware Scheduler for Hybrid Cloud
16mPreview: Cloud Computing
28mAI Workload Preemption in a Multi-Cluster Scheduling System at Bloomberg - Leon Zhou & Wei-Cheng Lai
24mReal-Time Streaming Segmentation on Azure
32mCase Study of Batch Processing With Spring Cloud Data Flow Server in Cloud Foundry
19mDShuffle: DPU-Optimized Shuffle Framework for Large-scale Data Processing
20mEnergy Efficient Placement of Edge Workloads
41mRunning Apache Flink everywhere