Move Fast and Meet Deadlines: Fine-grained Real-time Stream Processing with Cameo
summary
Cameo treats operator invocations as serverless functions and attaches priority context metadata to events so schedulers stay stateless while propagating deadlines; it computes a frontier time and estimates per-message execution cost to order work across DAG stages. A runtime priority-generation API lets jobs encode latency targets and query semantics. On Azure experiments Cameo cuts median query latency by 2.7× single-tenant and 4.6× multi-tenant, handles transient spikes, and implements token-based rate shares (20/40/40) via prioritized messages.
AI-written summary · about
chapters (7)
- 0:00Cameo introduction and goals
- 2:46Stream processing resource challenges
- 3:49Deadline-aware scheduling in Cameo
- 7:43Cameo scheduling mechanism
- 9:25Priority context propagation
- 10:48Programmable scheduling in Cameo
- 11:50Conclusion and future
related talks
30mFive Challenges in Real-time Stream Processing and Five Solutions
42mBuild a Real-Time Event-Driven Java Solution in Azure
48mTaming & Composing High Performance Stream-Oriented Processing Systems
39mReal-Time Event Processing and Handling Stateful Applications on Apache Mesos
19mBuilding Stateful Streaming Pipelines
24mReal-Time Streaming Segmentation on Azure
39mDeploying Fast Data Pipelines
36mMoving Faster: Why Intent Media Chose Cascalog