Hacks to Make LLM Training Faster
summary
Six practical wins: switching from float32 to float16/BF16 yields roughly 5x speedups on common GPUs; quantizing to int8 keeps accuracy while int4 halves model “knowledge” unless combined with LoRA fine‑tuning; offloading activations to system RAM asynchronously (Unsloth checkpointing) adds ~1–2% time overhead but cuts GPU memory; use tensor cores and FlashAttention, fuse kernels and TorchCompile, prefer T4s over P100s for inference, apply chunked cross‑entropy and chain matrix bracketing to lower FLOPs, and focus on higher‑quality web data (FineWeb).
AI-written summary · about
related talks
01h 18mOptimizing LLM Training and Inference Performance on GPUs
24mOptimizing CPU LLM Inference in PyTorch: Lessons From VLLM
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
24mLLM Inference Optimization From Simple To Sophisticated
28mDatabricks' vLLM Optimization for Cost-Effective LLM Inference
24mCost and Performance Optimization of LLM Inferencing.
16mHow fast are LLM inference engines anyway?
19mNo GPU Left Behind: Scaling Online LLM Training With...