Optimal on Paper, Broken in Reality
summary
Shows a concrete method to temper optimistic benchmarks by plotting accuracy versus training size and exposing how grid-search numbers hide sample-size effects, demonstrates that universal sentence encoder barely beats count vectors once you zoom and add data, compares fastText and logistic regressors, finds mislabeled examples in the GoEmotions Reddit set using cleanlab heuristics, and ships a new Python toolkit doubt lab that ranks suspicious rows so noisy labels get inspected rather than trusted.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:30Benchmarking embeddings for chatbots
- 4:21Uncertainty in model comparisons
- 5:42Include simple baselines
- 7:05Data volume impact
- 8:28Benchmarks without context
- 10:18User needs vs model optimization
- 12:25Label errors in sentiment data
- 15:10Detecting mislabeled examples
- 17:38Uncertainty for label errors
- 20:06High-bias vs embedding approaches
- 21:51Label errors threat
- 23:55Illusion of reliability
- 24:56Data quality focus
- 25:39Lessons beyond NLP
- 27:00Doubt Lab for label errors
- 28:47Q&A
related talks
30mMachine Learning: Beyond the Tutorial
54mKeynote: Can you trust your (large language) model?
01h 05mCan you trust your (large language) model?
30mData-Centric AI Cookbook-let's prep that data
19mHow to look at your data
01h 04mCan you trust your (large language) model?
45m"Breaking black-box machine learning"
28m🧼 From GPU-poor to data-rich: data quality practices for LLM fine-tuning