EfficientBERT: Trading off Model Size and Performance
summary
Distills a fine-tuned BERT teacher into smaller student models on SQuAD 2.0, seeding students with DistilBERT weights and running a simultaneous architecture search and hyperparameter sweep via SigOpt + Ray multimetric Bayesian optimization across 20 EC2 nodes. Baseline is 66M parameters, 67% accuracy; the Pareto frontier surfaces smaller models that match accuracy and a top model at ~70% accuracy. Findings: prefer no dropout, weight soft-target loss, and compress by reducing layers more than pruning heads; metric threshold 50 prevents trivial unanswerable guessing.
AI-written summary · about
chapters (16)
- 0:00Introduction and overview
- 2:14Compressing BERT overview
- 3:41Use case and dataset
- 5:08Distillation process explained
- 7:14Defining student model
- 7:55Baseline model results
- 8:36Optimization process setup
- 10:03Handling unanswerable questions
- 11:10Tuning parameters overview
- 12:13Ray and SigOpt integration
- 13:19Pareto frontier results
- 14:43Optimal model architectures
- 15:25Experiment dashboard analysis
- 16:35Best performing model details
- 17:59Error analysis and insights
- 20:03Key takeaways and conclusion
related talks
24mEfficient BERT: Optimal Multimetric Bayesian Optimization
08mLeveraging large-scale models for efficient learning
01h 00mServing Pytorch Models in Production
28mFine-Tuning Small Language Models with Python
44mTransfer learning in NLP
01h 38mTraining Large Language Models: Practices and Research Questions
28mIndustry NLP: Multitask & Continual Learning
22mSpecializing Small Language Models with less data