Symbiotic Relations between Decoupled Training, Optimization, and Federated Learning
summary
Scales DiLoCo across Chinchilla-optimal token horizons and finds concrete rules: inner optimizer is AdamW, outer optimizer is SGD with Nesterov, sync every H steps across M replicas, and Muon can replace inner to yield MuLoCo. Surprising empirical claim: M=1 Lookahead+Nesterov outperforms standard data‑parallel across model sizes (confirmed by Meta’s SNOO). Reports larger models tolerate decoupling better, shows critical batch‑size fragility with larger M, and flags federated‑style heterogeneity as a potential tool when intentionally partitioned.
AI-written summary · about
chapters (14)
- 0:00Federated and large-scale optimization
- 1:10Communication bottlenecks
- 2:16Decoupled training as solution
- 3:25Optimization and decoupled trends
- 4:09DiLoCo method details
- 7:07Optimization advances for decoupled
- 9:35Scaling laws for DiLoCo
- 16:56Decoupled training impact
- 22:07DiLoCo and federated learning
- 28:26Heterogeneous data leveraging
- 34:09Federated optimization insights
- 37:05Decoupled training without compute
- 39:58Open questions and future
- 40:24Q&A
related talks
01h 00mFedOpt for LLMs
20mBeyond MLOps - How Federated MLOps Unlocks the Next Frontier of AI Innovation
01h 05mTutorial: Federated Optimization, Part III
32mOptimizing Machine Learning Training Infrastructure: A Governance Approach
46mSpeeding Up LLM Training by Over 45%
20mAccelerating Distributed MoE Training and Inference
42mFederated Learning: Unlocking the Values of Discrete Data for Enterprise
20mFwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed...