OPT-175B: LLM Development Lifecycle & Challenges
summary
OPT-175B is a 175 billion parameter GPT-style model released for research in May 2022 and requires access request; development faces repeated large-scale failures: frequent ECC and InfiniBand GPU errors causing ~90+ restarts, long debugging, and new cluster health checks (GPU burn-in, nickel tests). Training hits numerical instabilities from FP16 overflow spikes and gradient NaNs, solved by changing hyperparameters to GPT-3–style settings, clipping from 1.0 to 0.3 and lowering learning rate. Benchmarks over 14 tasks show a zero-shot gap to GPT-3 that largely disappears at 32-shot.
AI-written summary · about
chapters (8)
- 0:00Introduction to OPT-175B
- 1:13Model architecture scaling
- 2:07Data collection and ablation
- 4:31Hyperparameter tuning
- 6:56Infrastructure challenges
- 8:37Managing training instability
- 12:25Benchmarking and evaluation
- 14:32Open sourcing the model
related talks
19mSystem for Scaling Large Language Models
43mState of GPT
13mPreview: ML Training
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…
25mAggressive LLMs Optimization: Making Them Work on Tiny Devices
27mProductionizing Open Source LLMs
43mGPT-x, is it really worth it?
31mOvercoming Challenges in Serving Large Language Model