Training Arctic
summary
Arctic is a 480B hybrid sparse-dense MoE (10B dense + 3.6B expert params, 128 experts) trained in under three months across three stages on 3.5 trillion tokens. The stack uses Snowflake + Kubernetes + AWS P5 (8× H100 80GB, EFA 3.2 Tbps) with DeepSpeed, Megatron, and Hugging Face; techniques include ZeRO stage‑1 and expert parallelism. A job coordinator automates fault detection, node replacement, and checkpoint resume; over 100 data experiments plus an Arctic‑small validation run enable the released model and open‑source weights on Hugging Face.
AI-written summary · about
chapters (5)
- 0:00Introduction
- 3:54Arctic stack and model
- 9:57Data preparation
- 17:11Fault tolerance
- 20:43Future of Arctic
related talks
24mBetween Public Benchmarks and Business Needs
01h 19mSnowflake's AI research: A look inside our generative AI innovations
28mServing a Frontier Open Model with Ray & vLLM
54mDeep learning on AWS with NVIDIA: From training to deployment
36mAccelerating High-Performance Machine Learning at Scale i...
16mNous Research
28mScaling AI Infrastructure
26mSpeed up Your ML Workloads With Kubernetes Powered In-memory Data...