Tutorial: Federated Optimization, Part II
summary
DIANA removes the interpolation assumption by compressing gradient differences, yields near‑linear convergence without shared optimizers, and can be implemented communication‑free by keeping an H vector on server. EF14 is heuristic but EF21 fixes it, provably handling biased compressors (worse theory but broader robustness). MARINA compresses gradient differences with rare full sends, benefits from correlated/permutation compressors (theory shows up to √n or √d gains), and experiments plus Hessian‑compression proofs show large practical and local Newton‑type wins.
AI-written summary · about
chapters (11)
- 0:00Federated optimization intro
- 1:17DIANA without interpolation
- 8:23Error feedback origins EF14
- 15:08EF21 vs EF14 improvements
- 24:02MARINA for non-convex
- 33:11Correlated compression in MARINA
- 41:15Bidirectional compression benefits
- 45:41Local training fundamentals
- 53:19Fifth generation of local gradient descent
- 1:02:00ProxSkip and EF21 link
- 1:12:00Summary and Q&A
related talks
51mExploiting Similarity in Federated Learning
54mTight analyses of first-order methods with error feedback, in homogeneous and heterogeneous setups
16mTHC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression
01h 18mOptimization I
45mScale Learning and Reasoning Across Heterogeneous Gradients and Semantics
01h 01mFederated Learning in the Generative AI Era
45mSymbiotic Relations between Decoupled Training, Optimization, and Federated Learning
01h 07mOptimization Crash Course (continued)