Gold standard data: lessons from the trenches
summary
Three case studies show exactly what matters: concrete labels, tooling and measured outcomes. Word-embedding evaluation reveals huge human variance (ratings span 5–9 on a 10‑point scale) and an achieved 0.87 inter‑annotator agreement in one dataset; medical symptom annotation requires bespoke, behind‑firewall tooling and specialist annotators and takes a week to set up. Crowdsourcing via Mechanical Turk/CrowdFlower yields noisy clicks (pay ~1–10¢, ~60 simple items/minute) so continuous quality checks, golden tests and retraining are essential.
AI-written summary · about
chapters (6)
- 0:00Machine learning and data quality
- 3:35Garbage data ruins ML models
- 4:59Word embeddings case study
- 17:18Medical symptom annotation
- 27:50Crowdsourcing pitfalls
- 30:21Q&A
related talks
29mAnnotating data the right way
39mThe practice of acquiring good labels
34mScaling Trust
01h 11mACHIEVING SOCIAL JUSTICE FOR DATA WORKERS: IS THERE A ROLE FOR HARMONISED STANDARDS?
31mChallenges and Considerations in Language Model Evaluation
25mEmbracing Real-World Pluralism: Moving beyond "gold standard" truth in AI evaluation
37mOptimal on Paper, Broken in Reality
19mHow to look at your data