FlexOlmo: Open Language Models for Flexible Data Use
summary
Proposes FlexOlmo as a Mixture‑of‑Experts recipe where independent parties train per‑domain FFN experts and small router vectors locally, then FlexOlmo merges those weights into a single sparse MoE without joint retraining; experiments scale to 37B parameters (20B active) with experts of 7B, show up to 41% relative gain over public‑only pretraining and retain about 90% of a fully centralized MoE, while supporting opt‑in/opt‑out control.
AI-written summary · about
chapters (11)
- 0:00FlexOlmo intro and data fragmentation
- 7:16Federated learning limits
- 8:21FlexOlmo architecture and training
- 11:43Mixture of Experts basics
- 16:00Coordinating with public model
- 21:40Non-parametric router design
- 29:19FlexOlmo training recap
- 31:17Experimental setup and datasets
- 32:53Benchmark results
- 38:18Summary and open problems
- 40:31Q&A
related talks
58mFederated Open Language Models: Training LMs on Distributed Data
12mSmartMoE: Efficiently Training Sparsely-Activated Models through Combiningand...
31mVeRL: Extreme Optimization Practices for Ultra-Large-Scale MoE Models
46mSpeeding Up LLM Training by Over 45%
26mOpenFL (Federated Learning) Building better AI models with private data
20mFlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees
18mSmolLM: The Rise of Smol Models
25mOptimizing Reinforcement Learning at Trillion-Parameter Scale