System for Scaling Large Language Models
summary
Describes production lessons for serving multi‑terabyte LLMs: uses LoRA fine‑tuning to avoid full‑model forks, deploys NCCL topology‑aware collectives and CUDA graphs to cut communication and kernel launch overhead, and caches attention keys/values to speed generation. Notes GPT‑3 175B has 96 layers, 2048 token window, and ~two‑thirds of params in feed‑forward blocks so optimizing them yields big wins. Uses pipeline+tensor parallelism, powers‑of‑two batching, and a tokens_generated/total utilization metric to expose real headroom and improve throughput.
AI-written summary · about
chapters (11)
- 0:00Start
- 0:17Look back
- 4:53GPT-3 in Numbers
- 7:41Platform for large models
- 8:56MSCCL
- 9:30CUDA Graphs
- 10:23KV caching
- 10:45Model Partitioning
- 11:26Multi-node Serving
- 13:02LORA (Low-Rank Adaptation of LLMs)
- 14:41Traffic Routing
related talks
21mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
16mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
43mBeyond the Hype: A Realistic Look at Large Language Models
30mLarge Language Models and Prompt Engineering for Developers
23mOperationalizing Large Language Model Training Pipelines
44mHow to build your own Tiny Language Model from scratch
27mProductionizing Open Source LLMs
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM