AI Workload Preemption in a Multi-Cluster Scheduling System at Bloomberg - Leon Zhou & Wei-Cheng Lai
Talkat KubeCon + CloudNativeCon Europe 2025 · on Thursday, 2025-04-03· at 14:15· Level 1 | Hall Entrance S10 | Room B·
summary
Bloomberg implements Karmada priority and preemption to schedule tens of thousands of GPU training jobs across multiple on‑prem bare‑metal Kubernetes clusters, sourcing priority values from federated or Kubernetes priority classes and storing them on resource bindings. The system enforces Karmada priority and preemption with a single-scheduler preemption rule to avoid oscillating evictions, prefers evicting the fewest jobs that free required GPUs, runs in production, provides queue‑time estimates from historical averages, and uses local NVMe model caching and checkpointing recommendations.
AI-written summary · about
chapters (8)
- 0:00Bloomberg AI scheduling intro
- 1:25AI’s role at Bloomberg
- 3:54Bloomberg’s Kubernetes training
- 8:12Prioritizing AI workloads
- 10:39Carmada scheduling preemption
- 14:50How preemption balances efficiency and disruption
- 17:57Efficiency vs user experience
- 20:23Q&A
related talks
22mReal-Time GPU Job Scheduling Latency Prediction in Multi-Cluster Kubernetes
10mFair Scheduling for Deep Learning Workloads in Kubernetes
18mKeynote: Adapting Kubernetes for Machine Learning Workflows
33mFrom spreadsheet scheduling to Kubernetes: building an on-premise ML platform
28mMultitenancy and Fairness at Scale with Kueue: A Case Study
31mShared GPU Scheduling & Proactive Autoscaling: A Production Blueprint for 1000+…
18mKeynote: Optimizing AI Workload Scheduling: Bilibili's Journey To an Efficient Cloud Native AI Platform
20mBoosting Training and Inference Performance via Topology-Aware Scheduling of Heterogeneous Resources