Cluster Management for Large Scale AI and GPUs: Challenges and Opportunities
Talkat Cloud Native + Kubernetes AI Day Europe 2025 · on Tuesday, 2025-04-01· at 14:05· Level 1 | Hall Entrance N10 | Room G·
summary
Runs Kubernetes-native tooling—Prometheus, Grafana, DCGM exporter, GPU Operator, ML Batch/Q and AppWrapper—across multi-node H100 GPU clusters with NVMe and ~3.2 TB/s network and shows an automated stack that detects and responds to hardware faults (meta finds 78% of failures are hardware, ~60% GPUs; a paper reports >4,000 GPU-caused job crashes over 2.5 years). Built autopilot health checks that export Prometheus metrics and label nodes, integrated with Slack cluster queue quota adjustments and AppWrapper resets to evict, reprovision and steer workloads automatically.
AI-written summary · about
chapters (10)
- 0:00GPU cluster management intro
- 1:04Bella and Bella 2 specs
- 3:17AI workload lifecycle
- 6:28Fault tolerance essentials
- 10:07GPU monitoring with Prometheus DCGM
- 11:33Autopilot health checks
- 15:05Fault recovery with Q queues
- 17:51AppWrapper workload hardening
- 21:45Live fault recovery demo
- 23:27Key takeaways open-source tools
related talks
36mProduction GPU Cluster with K8s for AI and DL Workloads
09mLightning Talk: GPU-Scanner: Extending CNCF Observability for Multi-GPU AI Workloads
31mShared GPU Scheduling & Proactive Autoscaling: A Production Blueprint for 1000+…
01h 15mTutorial: Build, Operate, and Use a Multi-Tenant AI Cluster Based Entirely on Open Source
41mBuilding & Operating a Large-scale HPC AI Cluster on Kubernetes
31mKubernetes For GPU Powered Machine Learning Workloads In Academia
32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers
22mBeyond Single-Cluster Limits: Scaling GPU Workloads Across Kubernetes With…