Methods for Evaluation of Imperfect Captioning Tools by Deaf or Hard-of-Hearing Users at ...
summary
107 deaf and hard-of-hearing participants watch 12 videos with captions at 5%, 15%, 50% error rates and complete probes: yes/no error detection, Likert usefulness/accuracy scales, numeric accuracy estimates, and multiple-choice comprehension quizzes; reading skill is measured with WRAT. Multiple-choice comprehension discriminates caption quality best for low-literacy participants, while Likert scales show literacy bias (stronger readers give harsher ratings) and often fail to distinguish 5% vs 15% errors for weaker readers. Report participant literacy and choose probes by literacy level.
AI-written summary · about
chapters (14)
- 0:00Introduction to research
- 1:09Challenges with interpreters
- 2:11ASR errors and usefulness
- 3:15Prior evaluation methods
- 4:39Research questions and probes
- 6:23Study methodology details
- 7:38Caption quality levels
- 9:03Interpreting results methodology
- 10:29Discriminative ability findings
- 12:15Literacy bias in probes
- 13:41Recommendations and limitations
- 14:42Future research directions
- 15:26Closing and acknowledgments
- 15:48Q&A session
related talks
07mQuality Captioning as Universal Access...
10mBlind and Human: Exploring More Usable Audio CAPTCHA Designs
54mSounding out the web
59mAccessibility for the Deaf / Hard of Hearing and Progress Over Perfection
30mHear! Hear! An Accessibility Tale from a Hearing-Impaired Senior Software Engineer
30mHear! An Accessibility Tale from a Hearing-Impaired Senior Software Engineer
56mBeyond Colors and Captions: How to Provide More Inclusive Accessibility
06mToward User-Driven Sound Recognizer Personalization with People Who Are d/Deaf or Hard of Hearing