Understanding and Mitigating Gradient Flow Pathologies in PINN
summary
Diagnoses stiffness in the gradient flow of PINNs by measuring Hessian largest eigenvalues and backprop gradient histograms, showing imbalance between PDE residual and boundary/data terms that forces tiny learning rates (10^-6–10^-7) or hours of training. Proposes an online learning rate annealing that rescales per-term gradients using moment estimates, and a compact multiplicative U/V architecture with residual shortcuts. Across Helmholtz, Burgers, wave and Navier–Stokes inverse tests the combination improves L2 accuracy roughly 50–100x and eliminates boundary-fitting collapse.
AI-written summary · about
chapters (16)
- 0:00Introduction to speaker
- 1:14Physics informed neural nets
- 3:57Framework overview
- 6:12Historical context
- 8:21Burgers equation example
- 11:07Applications and limitations
- 13:01Pathologies in PINNs
- 15:18Gradient flow analysis
- 20:53Stiffness and loss landscape
- 26:22Adaptive learning rates
- 29:46Annealing algorithm
- 32:09Neural architecture updates
- 35:07Benchmark results
- 36:54Wave and fluid examples
- 39:15Conclusion
- 41:29Q&A session
related talks
41mOptimization Algorithms for Deep Learning
13mThe dynamical view of machine learning: implicit biases of large learning rates
01h 18mOptimization I
46mAnalyzing Optimization and Generalization in Deep Learning via Trajectories of Gradient Descent
04mNeural Networks Demystified: Part 5
52mSplitting Gradient Descent for Incremental Learning of Neural Architectures
01h 04mOptimization for Machine Learning II
59mOptimal Transport and PDE: Gradient Flows in the Wasserstein Metric