Understanding the end-to-end LLM training and inference pipeline
summary
Walkthrough of the pipeline names concrete choices and numbers: examines Llama 3.1 8B (128k vocab, 4,096 hidden size, 32 attention heads) and shows the memory math (8B params ≈ 16 GB in fp16, half that in fp8). Covers pretraining, supervised fine‑tuning, RLHF, and PEFT via low‑rank adapters, and measures latency with genperf metrics like time‑to‑first‑token and per‑token latency. Shows deployment stacks and tradeoffs using fp8/fp16 quantization and Triton and tzar llm for batching, KV‑cache sizing, and GPU selection.
AI-written summary · about
chapters (11)
- 0:00Welcome to LLM pipeline
- 1:24LLM terms core definitions
- 6:04Memory in LLM inference
- 9:13LLM inference workflow
- 12:39Tokenization and embeddings
- 17:18Attention and KV cache
- 21:34LLM training phases
- 23:23Low-rank adapters for fine-tuning
- 24:29LLM performance metrics
- 25:33NVIDIA tools for inference
- 27:19Q&A on attention and parallelism
related talks
38mOptimizing LLM Inference for the Rest of Us
22mTraining a LLaMA in your Backyard: Fine-tuning Very Large...
31mBuilding LLaMA: Meta's Director of GenAI
46mUnder the Hood of LLMs — all those questions you never dared ask about them
53mLLM Quality Optimization Bootcamp
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
30mLearning to wield large language models
30mLLM Inference Arithmetics: the Theory behind Model Serving