Integrating Constraints into Deep Learning Architectures with Structured Layers
summary
Claims that a weight‑tied, input‑injected network can be collapsed to a single implicit layer and trained by solving for its root, implemented as deep equilibrium models solved with Broyden’s method. Experiments replace a multi‑layer Transformer/TrellisNet block with a DEQ block of similar parameter count, match Transformer‑XL perplexity on WikiText‑103, cut memory by up to ~8× versus 60‑layer unrolling, and run about 2–3× slower at training and ~2× slower at inference.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 0:39Deep Equilibrium Models
- 1:52Depth
- 3:04Agenda
- 5:57Explicit Layers
- 7:50Implicit Layers
- 11:00Sat Optimization
- 12:51Implicit Function Theorem
- 15:50Takeaway
- 18:25Proposed Class of Structured Layers
- 20:21Weight Tied Input Injected Networks
- 21:29Expanding Depth to Capture Both Layers
- 25:25Deep Networks
- 27:14Summary
- 30:36Stacking Layers
- 33:51Do they exist
- 35:36Equilibrium point
- 40:01Residual point
- 42:01Sequence modeling
- 44:21Smallscale benchmarks
related talks
11mOn Implicit Regularization in Deep Learning
56mTutorial on Deep Learning II
39mTackling the Limits of Deep Learning
35mTowards Biologically Plausible Deep Learning: ...
37mMonadic Deep Learning
23mOn Implicit Regularization in Deep Learning
45mIntroduction To Deep Learning
28mNext-generation frameworks for Large-scale Machine Learning