ReLUs: An Alternative to the ReLU Activation Function
summary
Proposes a new activation that mixes sigmoid of sine and cosine to keep negative activations nonzero, adds one amplitude hyperparameter (~0.00025–0.0025), and ensures nonzero derivative so gradients don’t die. Implements it as sign-relu in Keras contribs, runs CNNs (MNIST, CIFAR-like ResNet), IMDB sentiment and Kaggle toxicity; shows small accuracy gains (few hundredths to a few tenths of a percent) but slower training (examples: MNIST 5:52 vs 7:43), experiments on MacBook and AWS and claims consistent improvement in many but not all runs.
AI-written summary · about
chapters (11)
- 0:00Introduction and motivation
- 3:11Career shift to AI
- 10:40Why create a new activation function
- 17:23Inspiration from neuroscience
- 21:17Industry challenges in ML
- 25:51Introducing SignReLU
- 30:52Neuroscience backdrop and references
- 35:18Datasets and experiments
- 38:22Performance comparison and benchmarks
- 40:18Future plans and porting to frameworks
- 42:11Q&A and closing
related talks
50mLearning and Generalization in Over-parametrized Neural Networks, Going Beyond Kernels
22mFormal Security Analysis of Neural Networks using Symbolic Intervals
27mDemystifying Deep Neural Networks
01h 04mNeural Networks meet Nonparametric Regression: Generalization by Weight Decay and Large...
01h 03mUnderstanding ReLU Networks Through Graph Problems and Polyhedra
01h 36mHow DeepRRP Works
32mResource Utilization as a Metric for Machine Learning
01h 14mApproximation Power