Simplifying Generative AI Model Training on Kubernetes using Helm Charts
summary
Claims a Helm-based stack that standardizes training by putting code and configs into an executable ConfigMap templatized via Helm and driving PyTorchJob/MP frameworks with small reusable charts, demonstrated across hundreds of runs and frameworks (Accelerate, Lightning+Ray, NVIDIA NeMo‑Megatron). Implements a Kubeflow Pipeline Helm-chart component that installs chart entries sequentially, mounts PVCs for shared checkpoints, and uses Karpenter for accelerator autoscaling. The artifacts (charts, values files, PVC pattern) are reusable, few in number, and validated end-to-end on Kubernetes.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:26Generative AI training with PyTorch
- 7:37CubeFlow training with PyTorch
- 10:30Training script challenges
- 12:19Helm charts solution
- 17:58Simple PyTorch training demo
- 21:33Nemo Megatron fine-tuning demo
- 26:08Resources and closing
related talks
05mProject Lightning Talk: Kubeflow Helm Chart
43mAt The Helm of Kubernetes: Repeatable Infrastructure Creation for Mere Mortals
37mAutomated Provisioning using HELM charts on Kubernetes
34mOne Chart to Rule Them All: Continuous Deployment with Helm at Ticketmaster
35mOptimizing Kubernetes Deployments with Helm
26mHelm your way with Kubernetes
41mHelm - the Better Way to Deploy on Kubernetes
27mDay 2 Operations of Helm Charts