Explainable modeling of annotations in crowdsourcing
summary
A joint aggregation and clustering probabilistic model is built (Dawid and Skene style aggregation + hidden‐variable mixture clustering) and learned with EM and Gibbs sampling to detect systematic labeling errors on a Mechanical Turk NER dataset. A retrospective experiment shows modest but consistent improvement in cluster quality versus aggregate‑then‑cluster, with gains growing as more data are added. A human study: ~100 participants finds clusters slightly improve sense‑making and prediction of future annotations but overall prediction accuracy remains low and some clusters are hard to interpret.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:13Crowdsourcing
- 1:18Crowdsource Aggregation
- 4:11Task: Named Entity Recognition
- 8:08Graphical Model
- 8:51Cluster Quality Experiment
- 10:14Human Experiment
- 14:10Conclusions
related talks
09mQuantitative Evaluation of Machine Learning Explanations: A Human-Grounded Benchmark
10mVOICE 2021 | Daria Baidakova, Toloka
32mEthical Human-in-the-Loop ML Pipelines
16mMaking recommendations explainable
22mThe Power of RAG at Scale: A Year of Explosive Growth and Observab...
56mDehumanizing Agents: Why Explainability is Crucial in the LLM Era
21mThe Exploratory Labeling Assistant: Mixed-Initiative Label Curation with Large Document Collections
31mAccuracy and Interpretability Through the Lens of Human-AI Teaming