Learning from Heterogeneous Sources 2026
2026-02-23..2026-02-27
Berkeley, United States
who it's for
federated learning researchers, machine learning optimization researchers, privacy engineers, distributed systems researchers, LLM engineers, reinforcement learning researchers, continual learning researchers, multi-task and multimodal researchers
topics
federated learning algorithms; client-drift variance reduction; communication-efficient compression with error feedback; differential privacy for decentralized training; exact unlearning via model merging; outer-optimizer Local SGD schedules; adaptive clipping optimizer Bi2Clip; algorithmic randomness privacy amplification; federated reinforcement learning bounds; personalized federated training; synthetic data for on-device LMs; robust mean estimation for heterogeneous users
| title | author | length |
|---|---|---|
| From the Ball-proximal (Broximal) Point Method to Efficient Training of LLM | Peter Richtarik | 52m |
Proposes the Ball-Proximal (Broximal) Point Method as a ball-constrained analogue of proximal methods and proves surprising guarantees: linear convergence and… | ||
| Scale Learning and Reasoning Across Heterogeneous Gradients and Semantics | Tian Li | 45m |
Proposes Bi^2Clip, a coordinate-wise bi‑directional clipping optimizer that approximates adaptive preconditioning without storing per‑client statistics (two… | ||
| "Federated Reinforcement Learning: Statistical and Communication Trade-offs | Yuejie Chi | 42m |
Analyzes federated Q-learning with rigorous sample and communication bounds: single-agent Q-learning needs roughly |S||A|(1/(1−γ))^4 / ε^2 samples; running K… | ||
| Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration | Ahmed Khaled | 31m |
Provides convergence guarantees for Local SGD showing the outer optimizer controls a tradeoff between optimization error and stochastic noise, and can… | ||
| The Many Faces of Heterogeneity: Federated, Continual, and Modular Learning | Marco Ciccone | 35m |
Claims that a single ridge-regression aggregator called Fed3R ridge regression recovers the centralized solution for linear probes and often outperforms… | ||
| Symbiotic Relations between Decoupled Training, Optimization, and Federated Learning | Zachary Charles | 45m |
Scales DiLoCo across Chinchilla-optimal token horizons and finds concrete rules: inner optimizer is AdamW, outer optimizer is SGD with Nesterov, sync every H… | ||
| Privacy Amplification from Structured Algorithmic Randomness | Ayfer Ozgur | 54m |
Claim: inherent algorithmic randomness in compression, model partitioning and structured subsampling measurably amplifies privacy so external Gaussian noise… | ||
| Federated, Synthetic, Personalized: Heterogeneity Here or There? | Zheng Xu | 55m |
Production on-device LMs use federated learning plus DP and secure aggregation to reach user-level epsilons mostly around 10 or lower and some under 1 by… | ||
| Privacy of Decentralized Machine Learning | Edwige Cyffers | 33m |
Shows that decentralization alone does not stop data reconstruction: honest-but-curious attackers that observe gossip messages can reconstruct private images… | ||
| Exact Unlearning of Finetuning Data via Model Merging at Scale | Kevin Kuo | 26m |
Proposes SIFT-Masks, an exact unlearning method that constrains fine-tuning to agree with a shared global sign vector so local task updates are sign-aligned or… | ||
| Privacy versus Robustness in Federated Learning: Limits and Algorithms | Youssef Allouah | 28m |
Proves a tight lower bound: any algorithm that is both locally differentially private and robust to an f fraction of malicious participants incurs an… | ||
| Exploiting Similarity in Federated Learning | Sebastian Stich | 51m |
A distributed proximal‑point framework (a modern DANE/CGM variant) adapts any centralized convex optimizer and shows the L smoothness term can often be… | ||
| Federated Learning in the Generative AI Era | Gauri Joshi | 01h 01m |
Proposes multi-headed LoRA (RAVAN) that freezes random A,B and trains small H heads so communication sends only H matrices; keeps 2.4M trainable parameters on… | ||
| FlexOlmo: Open Language Models for Flexible Data Use | Sewon Min | 46m |
Proposes FlexOlmo as a Mixture‑of‑Experts recipe where independent parties train per‑domain FFN experts and small router vectors locally, then FlexOlmo merges… | ||
| On continual learning with gradient descent for neural networks | Arya Mazumdar | 01h 00m |
Analyzes continual learning for a one-hidden-layer quadratic network trained by gradient descent on an XOR cluster with Gaussian noise (task means orthogonal)… | ||
| Not all data are created Equal: Robust Mean Estimation from Heterogeneous and Unreliable Users | Maryam Aliakbarpour | 47m |
Model where N users each send n samples, an epsilon fraction of users are fully adversarial and the remaining users are “good” but lie within a proximity alpha… | ||
| A Complex Picture of Multi-task Learning | Samory Kpotufe | 47m |
Multitask learning with N source datasets each of size n yields a mixed picture: an oracle vs adaptive gap where an oracle that knows which M sources are close… | ||
| Learning from multiple modalities, Predicting on unseen tasks | Zaid Harchaoui | 41m |
Zero-shot prediction claims to deliver strong downstream accuracy with no task-specific labels by training large contrastive or self-supervised models on… | ||
| Personalized Collaborative Learning with Affinity-Based Variance Reduction | Chenyu Zhang | 48m |
Proposes AffPCL, a personalized collaborative algorithm for multi-agent linear systems that uses bias correction and importance correction to handle objective… | ||
| The Statistical Fairness-Accuracy Frontier | Alireza Fallah | 45m |
Characterizes the fairness-accuracy frontier for two-group linear regression under finite samples: proves minimax-optimal estimators when covariate covariance… | ||
| Are We Measuring the Right Thing? Distribution Shift Lessons for Federated Learning | Sanmi Koyejo | 53m |
Large deployed sepsis model that reported AUCs ~0.76–0.83 falls to ~0.63 at one hospital, alerts on ~18% of admissions and only adds ~7% more sepsis detections… | ||
| Tight analyses of first-order methods with error feedback, in homogeneous and heterogeneous setups | Aymeric Dieuleveut | 54m |
Provides tight, sometimes numerically exact, convergence rates for compressed gradient descent, EF and EF21 by searching for optimal Lyapunov functions via… | ||
| nothing matches | ||