Improving GPU Utilization and Accelerating Model Training with Scheduling Framework and NRI
summary
Describes how ByteDance uses Kubernetes scheduling framework and containerd’s Node Resource Interface (NRI) to manage GPUs: implementing GPU-sharing with fine-grained allocation (as small as 1% compute and 1 MiB memory) for AI inference to reduce waste, and adding topology-aware scheduling plus a custom GPU‑RDMA affinity strategy at the root complex level to enable GPUDirect RDMA and speed up distributed model training.
AI-written summary · about
related talks
20mBoosting Training and Inference Performance via Topology-Aware Scheduling of Heterogeneous Resources
38mMinimizing GPU Cost for Your Deep Learning on Kubernetes
38mMinimizing GPU Cost for Your Deep Learning on Kubernetes
28mGPU Reservations: Maximizing Utilization and Fairness Across Teams
41mPanel: AI Infra Best Practices: Enterprise Do’s and Don’ts
38mImproving GPU Utilization using Kubernetes
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs