Hydro: Surrogate-Based Hyperparameter Tuning Service in Datacenters
summary
Hydro builds a surrogate-based tuning service that trains scaled-down models via Torch FX, applies mu parameterization automatically, fuses multiple trials into one compiled workload, and opportunistically runs trials inside pipeline bubbles of large pretraining jobs. It reports up to 78.5x makespan reduction versus Ray Tune with no loss in final quality, over 20x throughput gains from inter-trial fusion on A100, and public source code on GitHub for replication.
AI-written summary · about
chapters (8)
- 0:00Hydro introduction and motivation
- 2:33Hyperparameter tuning efficiency challenges
- 3:58Surrogate-based tuning with Hydro Tuner
- 6:27Mu parameterization benefits
- 8:14Model scaling and fusion techniques
- 9:42Hydro Coordinator for data centers
- 12:53Evaluation and results
- 14:57Summary and open-source
related talks
35mHyperparameter Tuning Using Kubeflow
35mCreating Your Own LLM Tuning Platform with Open Source Technologies
25mHyperparameter Tuning and Visualization in Deep Learning
16mmTuner: Accelerating Parameter-Efficient Fine-Tuning on Multi-GPU Servers with...
38mA Guide to Modern Hyperparameters Turning Algorithms
37mTraining and Optimisation of Large Transformer Models: An ATLAS and CERN Use Case - Ricardo Rocha, CERN & Maxence Draguet, University of Oxford
27mDistributed Deep Learning with Horovod on Ray
24mHelion 1.0: A High-Level DSL for Performance Portable Kernels