How to Train an LLM with Argo Workflows and Hera
summary
Builds a reproducible Argo/Hera workflow that provisions a training etcd, spins up 4 nodes × 4 GPUs = 16 GPUs, mounts SSD-backed PVs and /dev/shm, and launches PyTorch’s torchrun using etcd rendezvous to run fully sharded data parallel training on LLaMA2 fine-tuning (llama-recipes script). Workflows treat runs as ephemeral, use exit handlers to delete etcd and let cluster autoscaler tear down GPUs, and rely on checkpointing and torchrun retries to recover failed ranks.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:36Outline
- 1:03Motivation
- 1:46Foundation Models
- 2:49Fine Tuning
- 3:43Infrastructure
- 4:44Architecture Diagram
- 6:10Distributed Key Value Store
- 7:03Workflow Steps
- 8:08Pythor Example
- 10:11Teardown
- 11:45Hera Code
- 16:06Training Workflow
- 18:14Dynamic Volumes
- 19:28Dependency
- 21:16Outro
related talks
33mAccelerating the LLM Lifecycle on the Cloud
27mEmpowering Autonomy: BYD's Journey Taming Million-Task Scale With Argo...
20mScaling Gene Therapy Research with Argo Workflows and Hera
27mYes You Can Run LLMs on Kubernetes
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
51mSpeeding up deployment of machine learning models
25mPanel: Argo for ML: Achieving Scalability and User Experience