Scaling GPU Clusters Without Melting Down!
Talkat KubeCon + CloudNativeCon Europe 2025 · on Wednesday, 2025-04-02· at 11:15· Level 1 | Hall Entrance S10 | Room A·
summary
Cluster API server hits OOM when two concurrent list calls fetch 5,000 large secrets (~0.2 MB each); mitigation is API Priority and Fairness limiting concurrent list requests to two. Tuning Go GC to GOGC=50 doubles GC frequency and drops live heap across control-plane components. Long‑lived HTTP/2 connection skew resolves by enabling goaway-chance, which rebalances within ~15 minutes. Scalability is validated with KWOK simulation across 132 clusters to 4,000 workloads: DRA halves scheduling latency and cuts control‑plane memory ~30% and API‑server footprint ~20% versus the topology-aware scheduler.
AI-written summary · about
chapters (12)
- 0:00Scaling GPU clusters
- 1:25Secrets list calls causing OOM
- 3:14API priority and fairness
- 5:00Memory leaks after restarts
- 6:05Go garbage collection tuning
- 9:59API memory skew failures
- 13:34Goaway chance parameter
- 17:02Secret handling mistakes
- 19:14Future DRRA changes
- 22:04Scale testing with Quark
- 25:42Key takeaways
- 28:09Q&A
related talks
22mBeyond Single-Cluster Limits: Scaling GPU Workloads Across Kubernetes With…
21mExperience Operating Large GPU Clusters at Organizational Scale
28mShared Yet Isolated at Scale: Building Multi-Tenant Inference Platform on…
43mIntro + Deep Dive: SIG Scalability
31mGPU Hunter: Architecting Global GPU Availability With MultiKueue
38mImproving GPU Utilization using Kubernetes
28mGPU sharing done right: Secrets, security, and scaling with Vault and vCluster
37mOne Cluster to Rule Them All - Multi-Tenancy for 1k Users