Active Teaching, Human Learning
summary
Empirical claim: active learning often helps but sometimes hurts depending on number of classes, tiny random starts, or huge batch sizes; synthetic make_classification benchmarks show gains are small and fragile. Built tools: an online incremental training flow to avoid costly retrains, a UMAP-based single-label cluster inspector and a scraping/classifier front page with approximate-nearest-neighbor retrieval. Results: cluster view finds mislabeled subsets, online updates keep annotation servers live, and a surprising threshold around ~2,000 examples flips LM vs local model performance.
AI-written summary · about
chapters (21)
- 0:00Active teaching and learning intro
- 1:24What active learning means
- 3:31Benchmarking AL assumptions
- 5:19When AL fails in practice
- 7:02Deploying AL obstacles
- 8:47Overfitting and batch size
- 9:29Incremental model training
- 11:56Online learning trade-offs
- 13:20Silent AL validation failures
- 15:51AL versus independent sampling
- 16:11From AL to active teaching
- 18:19UMAP trick for data quality
- 22:57UMAP to detect labeling errors
- 23:17Active teaching with Vincent’s Archive
- 27:03Active teaching beyond AL
- 28:51Collaborative annotation datasets
- 30:14Q&A
- 30:37Online vs AL experiments
- 31:43Model disagreement with foundational models
- 34:51Dataset size saturation
- 37:01Closing remarks
related talks
30mActive Learning and Human-in-the-Loop Machine Learning
01h 31mInteractive Learning of Classifiers and Other Structures
39mActive learning in the interactive python environment
51mEfficient Deep Learning with Humans in the Loop
40mHuman Learning
45mActive learning and weak supervision in NLP projects
30mMachine Learning Techniques for Class Imbalances & Adversaries
36mWhen the Only Way to Scale up, is to Scale Down