FedOpt for LLMs
summary
Adaptive federated optimization applied to LLM pretraining uses JAX/DrJAX primitives and a non-blocking fragment sync to shard models (e.g. ~100 fragments) and run AdamW on workers with an outer SGD/Nesterov step and S4-style momentum (AdamW inner / SGD-Nesterov outer). Experiments compare to NanoDOH baselines, fully tune hyperparameters, and show near data-parallel FLOP-for-FLOP quality while cutting bandwidth by about two orders of magnitude (or to ~1% in some configs), results replicate.
AI-written summary · about
chapters (8)
- 0:00FedOpt for LLMs
- 3:21Federated optimization matters
- 9:03Distributed LLM training
- 16:04Federated meets LLM training
- 21:55Algorithms for distributed compute
- 28:52Scaling DiLoCo
- 33:25DiLoCo vs data parallel
- 45:13Q&A on DiLoCo
related talks
45mSymbiotic Relations between Decoupled Training, Optimization, and Federated Learning
42mFederated Learning: Unlocking the Values of Discrete Data for Enterprise
20mFwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed...
20mFlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees
21mAssyLLM: Efficient Federated Fine-tuning of LLMs via Assembling Pre-trained Blocks
51mExploiting Similarity in Federated Learning
30mLLMs: Beyond the Hype - A Practical Journey to Scale
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone