Active Learning and Human-in-the-Loop Machine Learning

on YouTube ↗30m

" summary

Argues that model gains come more from larger and cleaner labeled datasets than from tweaking algorithms, showing a toy result where error falls from ~13% at 10,000 records to ~8% at 20,000 and ~4% at 40,000 records. Reports noisy labels (feeding 90% accurate labels) hurting classifiers and cites a Kaggle search-relevance contest that climbs from ~30% to ~60% in days then stalls just under ~70%. Promotes active learning and human-in-the-loop labeling to cut labeling cost and rapidly improve accuracy.

AI-written summary · about

" chapters (12)

1. 0:00Introduction and audience check
2. 1:00Speaker background and CrowdFlower
3. 2:24Algorithm improvements vs feature engineering
4. 5:52Effect of more training data
5. 7:36Impact of clean data on accuracy
6. 10:55Debate on data cleaning and plateauing
7. 15:07Data cleaning and human-in-the-loop
8. 16:30Crowdsourced data collection examples
9. 21:01Public data sets and Kaggle competitions
10. 23:51Human-in-the-loop computing and Active Learning
11. 26:40Real-world applications of human-in-the-loop
12. 28:50Q&A and closing remarks

" related talks

• 37mActive Teaching, Human LearningVincent D Warmerdam · PyData Eindhoven 2023
• 51mEfficient Deep Learning with Humans in the LoopZachary Lipton · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 39mActive learning in the interactive python environmentJan Freyberg · PyData London 2019
• 37mBest Practices for Building AI DatasetsBradley Arsenault · Big Data Conference Europe 2019
• 31mReliable Data for Large ML Models: Principles and PracticesMary McGlohon · SREcon23 Europe/Middle East/Africa
• 33mKeys to Data-Centric AI: Asking AI to Improve its own Dataset.Jonas Mueller · Scale By the Bay 2023
• 29mWhy Deep Learning is driving demand for Human in the LoopKeith McCormick · Global Artificial Intelligence Virtual Conference 2022
• 37mMachine Learning, The Right WayVitaly Gordon · AI By the Bay 2017