Tutorial: Federated Optimization, Part I
summary
Overview of concrete results: the lecture compares cost models (notably the communication-to-server-only cost model), analyzes DCGD with unbiased compressors and shows Rand-1 sparsifier is optimal among Rand-k for that model, and proves compression can reduce wall-clock communication by up to factors like N or D. It introduces natural quantization (randomized power-of-two rounding) with provable low variance, shows biased top-K can break naive DCGD (needs variance reduction or AB correlated compressors), and gives a nine-dimensional counterexample to LoRA plus a randomized fix.
AI-written summary · about
chapters (6)
- 0:00Federated optimization intro
- 1:54Field overview and scope
- 8:08Cost models
- 23:57Algorithmic tricks
- 44:00Compression fundamentals
- 1:04:01Advanced compression
related talks
01h 08mData Driven Optimization Models and Algorithms
51mExploiting Similarity in Federated Learning
01h 07mIndustry Presentations
01h 14mDistributed and concurrent optimization for machine learning
28mPrivacy versus Robustness in Federated Learning: Limits and Algorithms
45mCharting the Landscape of Memory/Data Tradeoffs in Continuous Optimization: ...
55mZeroth-Order Methods for Convex-Concave Minmax Problems: Learning from Strategically Generated Data
01h 01mRobustness, Stochastics, Uncertainty 2