Exploiting Similarity in Federated Learning
summary
A distributed proximal‑point framework (a modern DANE/CGM variant) adapts any centralized convex optimizer and shows the L smoothness term can often be replaced by a much smaller delta replaces L similarity constant, yielding convergence in as few as 3–5 communication rounds when clients are nearly identical. Subproblem solves use an optimistic look‑ahead to remove log factors; nonconvex partial‑participation uses variance‑reduced estimators (SVRG/SARAH hybrids) and provably needs only about sqrt(N) clients per round to match full‑participation rates.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 2:12Federated learning optimization
- 7:11Communication challenges
- 9:01Local SGD limits
- 13:39Reducing communication
- 17:39Measuring similarity
- 21:08Why local methods work
- 24:35Composite gradient methods
- 27:39Optimization methods vs delta
- 30:38Optimistic reference points
- 32:46Server to device shift
- 33:56Experimental validation
- 35:33Client sampling
- 39:59Non-convex optimization
- 45:20Optimization targets
- 47:22Federated optimization vs learning
- 48:31Q&A
related talks
01h 05mTutorial: Federated Optimization, Part III
01h 13mTutorial: Federated Optimization, Part II
01h 00mFedOpt for LLMs
20mFwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed...
42mLearning Together, Distributed: An Introduction to Federated Learning
01h 12mTutorial: Federated Optimization, Part I
16mDP-BREM: Differentially-Private and Byzantine-Robust Federated Learning...
42mFederated Learning: Unlocking the Values of Discrete Data for Enterprise