SPADE: Signal-Aware DAG Scheduling and Dynamic Provisioning for Data Processing Clusters
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
SPADE jointly schedules DAG tasks and provisions resources using a relative importance metric derived from Decima/Graphene scores and an exponential threshold filter to decide which tasks to run or defer under time-varying signals like carbon intensity. Prototype on Spark over a 100‑node Kubernetes testbed and simulations on TPC‑H and Alibaba DAGs show SPADE cuts the signal-driven cost (carbon footprint) by 32.9% versus baselines while bounding makespan via a tunable gamma and provable worst‑case stretch.
AI-written summary · about
chapters (9)
- 0:00SPADE introduction
- 2:24Signal-aware DAG scheduling
- 4:52Modern DAG schedulers
- 6:17Relative importance metric
- 8:25Exponential threshold filter
- 9:30Joint scheduling and provisioning
- 10:53Evaluation setup
- 12:58Carbon intensity results
- 14:43Summary
related talks
29mDynamic Large Scale Spark on Kubernetes
35mDagobah, a Data centric Meta scheduler
31mDynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo Workflows and Argo Events
14mSTACD: STAC Extension with DAGs for Geospatial Data and Algorithm Management
29mDagster: a Framework for Data Processing Applications
43mNo shard left behind: Dynamic work rebalancing in Apache Beam
15mOptimizing Query Performance in Big Data Systems
52mSolving Analytical Problems using Apache Spark