Efficient Deep Learning with Humans in the Loop
summary
Argues active learning can cut labels but with caveats: a convolutional NER pipeline runs several× faster and using dropout-based Bayesian uncertainty or Bayes-by-backprop sometimes reaches 99% of performance with ~25% samples on certain datasets, yet benefits vanish or reverse depending on choices like GloVe vs Word2Vec and retraining strategy. Proposes smarter annotation: ask binary questions up the WordNet hierarchy to turn one image into partial labels, and finds empirically single noisy labels beat high redundancy for a fixed budget using iterative worker-noise reweighting.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:40Outline
- 1:30Deep Learning
- 3:18Strategies to cope w. Data Scarcity in DL
- 5:16Data Augmentation
- 6:35Transfer Learning
- 10:53Deep Active Learning Considerations
- 13:37Acquisition functions
- 15:09The Dropout Method (Gal 2017)
- 15:50Bayes-by-Backpropagation (weight uncertainty)
- 17:19Optimizing variational parameters
- 17:53Efficient Exploration for Dialogue Policy Learning w. BBQ-Networks
- 19:11Efficient Exploration - Thompson Sampling
- 20:46Deep Active Learning for NER
- 21:56Modeling - Encoders Word embedding
- 22:51Tag Decoder
- 23:10Active learning heuristics
- 25:25Reality Check
- 28:03How Transferable are Active Sets Across Learners? w David Lowell & Byron Wallace, under submission
- 35:53Active learning with partial feedback
- 38:14Three sampling strategies
- 39:34Quantitative results
- 42:01Classical Crowdsourcing Setup
- 47:56Annotation Platform
- 48:38Important features (original data)
related talks
29mWhy Deep Learning is driving demand for Human in the Loop
30mActive Learning and Human-in-the-Loop Machine Learning
45mActive learning and weak supervision in NLP projects
39mActive learning in the interactive python environment
37mActive Teaching, Human Learning
31mKeynote Q&A – Barbara Plank
18mActive Learning for Neurosymbolic Program Synthesis
43mADAM - Bootstrapping a Deep Neural Network Sequence Labeling Model with minimal labelling