Efficient BERT: Optimal Multimetric Bayesian Optimization
summary
Distillation of BERT pairs with SigOpt multimetric Bayesian optimization and Ray on 20 EC2 instances to search architectures and hyperparameters jointly; baseline DistilBERT is 66M parameters, 67% SQuAD2 exact-match and the optimized Pareto frontier includes a smallest model matching baseline and a best model at 70% accuracy. Optimization prefers reducing layers over pruning attention heads, low/no dropout and heavier soft-target loss, and uses a 50% metric threshold to avoid trivial unanswerable predictions.
AI-written summary · about
chapters (5)
related talks
21mEfficientBERT: Trading off Model Size and Performance
26mAddressing Some of Challenges When Optimizing Long-to-Train-Models
18mAlexandra Johnson, Software Engineer, SigOpt, NYC 2017
18mBest Practices for Hyperparameter Optimization
41mOptimizing Machine Learning Models
24mUsing Bayesian Optimization to Tune Machine Learning Models
17mOptimized Image Classification on the Cheap
28mIndustry NLP: Multitask & Continual Learning