Road to 5 Million Tokens: Breaking Barriers in Long Context Training
summary
Claim: a training stack plus a novel twist pushes context to 5 million tokens on commodity 8×H100 nodes. Specifics: fully sharded data parallelism, DeepSpeed Ulysses style context parallelism yielding an ~8x activation reduction, activation checkpointing, CPU offload for transformer inputs, and chunked sequence training. Novel build: Untied Ulysses further shards attention head groups to reuse buffers across iterations; results match memory‑optimized baselines at 8B/32B and extend sequence length ≈25% beyond prior Ulysses.
AI-written summary · about
chapters (9)
- 0:00Together AI introduction
- 1:40Long context training importance
- 3:47Memory bottlenecks
- 5:58DeepSpeed Ulysses parallelism
- 8:08Activation checkpointing and CPU offloading
- 10:14Untitled Ulysses optimization
- 12:41Results and trade-offs
- 13:22Key takeaways
- 14:03Q&A on attention complexity
related talks
49mBen Hall & Barbara Fusinska
15mStop Stuffing Your LLM's Context Window
49mContext is King
12mThe Ultra-Scale Talk: Scaling Training to Thousands of GPUs
01h 00mThe Frontier between Retrieval-augmented and Long-context Language Models
31mBuilding LLaMA: Meta's Director of GenAI
51mLong context modeling and generalization: two perspectives
27mThe Four Pillars of Large-Scale Decentralized Pre-Training: Methods, Bottlenecks...