SonicASL: An Acoustic-based Sign Language Gesture Recognizer Using Earphones
summary
A wearable system uses commodity earphones with a front-facing speaker and microphone to emit ultrasound and capture reflections, sending the audio to a phone that builds 2D Doppler spectrograms, cleans them with bandpass/bandstop and Gaussian/linear filters, and feeds them to an LSTM sequence model with a collapse loss to produce word sequences. Evaluation with N=8 on 42 words (20 reps each) and 30 sentences (12 reps) achieves 93.8% word accuracy and 90.6% sentence accuracy across indoor and outdoor scenarios.
AI-written summary · about
related talks
06mmmASL: Environment-Independent ASL Gesture Recognition Using 60 GHz Millimeter-wave Signals
10mSignGlass: First-Person View Comprehensive and Generalizable ASL Translation Using Wearable Glass
35mHow deep learning can help interpret American Sign Language
10mTeethTap: Recognizing Discrete Teeth Gestures using Motion and Acoustic Sensing on an Earpiece
27mIndian Sign Language Recognition(ISLAR)
09mBeyond Captions A Practical Pipeline for Live AI-Generated Sign Language Avatars
23mDesigning for Accessibility
57mApplied AI and Accessibility to Play Games in New Ways