Neural Networks Demystified: Part 5
summary
Explains a practical method to verify backprop gradients by computing a numerical gradient check: perturb each weight by a small epsilon perturbation, evaluate cost at plus and minus epsilon, and form the slope; repeat for every weight to build a numerical gradient vector and compare it to the coded gradient using the norm of the difference over the norm of the sum. Typical pass threshold is about 1e-8, and matching vectors confirm gradients are implemented correctly.
AI-written summary · about
related talks
52mNeural Networks Walkthrough
55mUnderstanding and Mitigating Gradient Flow Pathologies in PINN
59mDifferential Verification of Deep Neural Networks
20mIncremental Verification of Neural Networks
42mTowards Verification of General Neural Networks: A Dual Approach
52mSplitting Gradient Descent for Incremental Learning of Neural Architectures
25mIntroduction to Neural Networks
41mOptimization Algorithms for Deep Learning