Training LlaMa - A Storage Perspective
summary
GenAI checkpointing creates spikes that saturate storage in under 10 ms, with observed checkpoint growths of 60x–70x and an eight‑order‑magnitude range (~4,000x) that once dropped goodput to ~5%. Response: build dedicated AI clusters, wrap tectonic FS with a fuse wrapper to behave like NFS, replay workloads with garbage data, and add control delay plus adaptive LIFO IO scheduling. Outcome: restored cluster goodput, improved P99 success latency, avoided hotspots, and predictable horizontal scaling by adding nodes.
AI-written summary · about
chapters (15)
- 0:00AI research growth and storage
- 1:10Warehouse vs AI storage needs
- 2:12LLaMa training storage demands
- 3:37Checkpointing hardest for storage
- 4:44Developer expectations vs storage limits
- 6:06Off-the-shelf solutions for LLaMa 3
- 7:31AI industry speed flips storage
- 8:54Sumit’s role in AI storage
- 9:57Checkpointing spikes and isolation gaps
- 11:22Recharacterizing IO for AI workloads
- 12:27Debugging goodput drops
- 14:13Improving throughput with delay and LIFO
- 15:18Side effects in AI clusters
- 16:21Flow control for AI traffic
- 16:43Handling AI storage spikes
related talks
19mScaling AI Network with DSF
22mScaling Llama4 Training to 100K
21mDC Networks for Generative AI
34mScaling AI Like a Pro-PepsiCo’s LLM Deployment Strategy on K...
29mThe Cat and Mouse Game of Infra Cost vs. Latency
09mTransform Your Kubernetes Cluster Into a GenAI Platform: Get Ready-t...
19mServerless inferencing: an infrastructure point of view
42mFine-tune and deploy LLMs from Hugging Face (AIM304)