Quality Control in the Crowd: Inside and Out
summary
Discusses quality control for crowdsourced work on platforms like Yandex.Toloka, explaining that crowdsourcing differs from outsourcing because workers often try to cheat. Describes built-in Toloka tools for checking task results and addresses how workers learn to bypass those controls, implying the talk covers practical measures to detect and mitigate such cheating.
AI-written summary · about
chapters (10)
- 0:00Introduction to crowdsourcing
- 1:08Crowdsourcing experience
- 3:17Training platform workers
- 5:23Control tasks and honeypots
- 8:21Overlapping tasks for quality
- 10:09External quality tools
- 12:33Bot mitigation
- 15:03Data errors analysis
- 16:50Data labeling control
- 17:55Q&A
related talks
59mLessons learned from mobile crowdtesting
59mLessons learned from mobile crowdtesting
13mCommitting to quality
19mHow can we improve quality on a project?
33mReplace Human in the Center of the Software Quality
52mNavigating Software Quality - From Chaos to Control
35mTesting != Quality
56mControlling Quality in Software Development