Hyperparameter Tuning and Visualization in Deep Learning
summary
Practical recipe for automated tuning: use Weights & Biases Sweeps with Ray Tune integration to run random, Bayesian, and population-based searches, add Hyperband early stopping to cut compute by about 10x while matching best-model performance, and apply Bayesian+GP surrogate models so uncertainty guides the next trials. Concrete examples include GAN digit generation and semantic segmentation; measure parameter importance with a random-forest metric, inspect parallel-coordinates to find low-dropout/high-FC regimes, and start by overfitting a single batch and running short sweeps.
AI-written summary · about
chapters (23)
- 0:00Intro
- 1:11What are hyperparameters?
- 1:50Example hyperparameter tuning
- 3:02Data pipeline hyperparameters
- 4:32Why tune hyperparameters?
- 6:07How To Tune Hyperparameters
- 7:52Build intuition and prioritize your experiments
- 9:24Algorithmic hyperparameter tuning
- 9:48Grid search
- 11:03Bayesian Hyperparameter Tuning
- 11:41Population Based Training
- 13:15Surrogate Function (Linear Regression)
- 13:43Surrogate Function (Fancier Model)
- 14:06Surrogate Function (Gaussian Process)
- 15:23Hyperband Early Stopping
- 16:55Training a GAN
- 17:34Random Search
- 19:22Bayesean Optimization with Asynchronous Hyper Band Early Stopping
- 21:05Parameter importance of larger search
- 21:28Parallel Coordinates: Explore Correlations
- 22:21General advice
- 23:00Addressing underfitting
- 23:31Don't blindly optimize a single metric
related talks
38mA Guide to Modern Hyperparameters Turning Algorithms
18mBest Practices for Hyperparameter Optimization
26mModel Selection with Python: An Introduction to Hyper Parameter Tuning
35mHyperparameter Tuning Using Kubeflow
16mHydro: Surrogate-Based Hyperparameter Tuning Service in Datacenters
23mTotal Tidy Tuning Techniques
18mAlexandra Johnson, Software Engineer, SigOpt, NYC 2017
38mCutting Edge TensorFlow: New Techniques