Optimizing LLM Training and Inference Performance on GPUs
summary
Workshop shows hands on use of H100 GPUs with Docker and redeemable H100 credits ($30 for ~1.5h) and demonstrates concrete wins: tensor cores cut 64 scalar FMAs to one MMA, FP8 mixed precision yields ~2.8× matrix throughput and ~2× training throughput in examples, and speculative decoding reports up to ~5.6× token generation speed. Covers Transformer Engine, VLM paged KV cache to avoid fragmentation, and prefix-sharing trees (AshiLane) for reuse, with real code exercises that run and reproduce the reported numbers.
AI-written summary · about
chapters (14)
- 0:00GPU optimization workshop intro
- 2:53AI and GPU timeline
- 8:03GPU vs CPU architecture
- 14:10Tensor vs CUDA cores efficiency
- 22:21Quantization trade-offs in training
- 26:28Mixed precision training
- 31:04Parallelism for large models
- 32:30Inference optimization basics
- 37:56KV cache with vLLM
- 44:31Speculative decoding
- 51:53Tensor core demo
- 58:41Transformer engine training
- 1:06:13vLLM setup and benchmark
- 1:17:01Workshop summary
related talks
08mHacks to Make LLM Training Faster
26mEfficiently serving LLMs at scale.
26mAmazingly Fast and Incredibly Scalable Inference...
15mOptimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
31mThe Evolution of Multi-GPU Inference in vLLM
19mNo GPU Left Behind: Scaling Online LLM Training With...
22mWhich GPU for Local LLMs?
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM