Tutorial: Federated Optimization, Part III
summary
ProxSkip connects compression plus EF21 to acceleration through local training and links to proximal-point lineage (DANE, Axfeldt, double stabilization) while device heterogeneity uses per-worker compute times tau_i and unbiased SGD with bounded variance to target stationary points. Key algorithms: Rennala SGD uses a magic mini‑batch size ~sigma^2/epsilon (ignores very slow workers so harmonic mean of taus appears), Shadowheart SGD sets an implicit equilibrium time T and compresses buffered sums, and Ringmaster SGD* enforces a staleness buffer R (accept or drop updates) to get time‑optimal, asynchronous guarantees.
AI-written summary · about
chapters (15)
- 0:00ProxSkip and local training
- 3:14Device heterogeneity and costs
- 10:41Data heterogeneity and worker selection
- 13:30Asynchronous SGD time complexity
- 20:51Rennala SGD for compute
- 25:28Mini-batch size in Rennala SGD
- 31:04Homogeneous regime comparison
- 33:46Malania SGD for data heterogeneity
- 34:09Shadowheart SGD with comms
- 40:28Freya PAGE for finite sums
- 41:16Ringmaster SGD on networks
- 46:11Sync vs async optimality
- 51:39Ring Leader SGD for data
- 52:00Open problems in federated
- 1:04:53Q&A
related talks
51mExploiting Similarity in Federated Learning
17mZEN: Empowering Distributed Training with Sparsity-driven Data Synchronization
45mSymbiotic Relations between Decoupled Training, Optimization, and Federated Learning
45mScale Learning and Reasoning Across Heterogeneous Gradients and Semantics
01h 07mOptimization Crash Course (continued)
47mThe Importance of Better Models in Stochastic Optimization...
33mStochastic Optimization for Big Data Machine Learning Problems
58mFirst-Order Stochastic Optimization