Sculpting Subspaces: Constrained Full Fine-Tuning for Continual Learning in LLMs
summary
A full-parameter continual-learning method leverages adaptive SVD to split each weight matrix into a frozen high-rank subspace and a tunable low-rank subspace, then constrains updates with projected gradient descent so gradients are orthogonal to preserved directions. Evaluated on T5-large and LLaMA‑27B and on the TRACE benchmark, it matches or exceeds baselines, reports up to 7% higher accuracy versus orthogonal LoRA, retains general abilities and shows modest backward transfer.
AI-written summary · about
chapters (12)
- 0:00Continual post-training in LLMs
- 2:29Catastrophic forgetting in models
- 5:19Approaches and trade-offs
- 6:41Constrained full fine-tuning
- 10:56Problem setup and methodology
- 13:46Retention ratios and subspace splitting
- 16:32Training with gradient projection
- 18:19Evaluation on benchmarks
- 23:16TRACE benchmark and metrics
- 25:45Results on abilities and safety
- 27:53Vision and limitations
- 28:57Q&A
related talks
01h 12mRecent Progress in High-Dimensional Learning
29mLocal LLM fine-tuning: a practical example
37mLinear representations of concepts in modern AI models
01h 06mAnalytic Methods for Supervised Learning III
01h 05mOptimization for Machine Learning I
01h 27mChallenges in Making LLMs Safe and Robust
21mScaling LLM Inference with vLLM
26mVLLM-Helion: SOTA LLM Performance by Advanced Autotuning and Fine-grained Dispatching