SoundLip: EnablingWord and Sentence-level Lip Interaction for Smart Devices
summary
SoundLip uses a phone’s speaker to emit ultrasound sensing and the microphone to capture reflections from moving lips, then preprocesses phase and amplitude (IQ data, multi-frequency) and removes static multipath via gradients. Word-level uses a Hierarchical CNN (separate phase/amplitude CNNs plus merged CNN) and sentence-level uses a GRU encoder-decoder with combined CTC+attention. Evaluation on 20 words and 70 sentences from 12 participants yields 91.2% word accuracy and word error rate 7.1%, running in real time and robust to noise, distance, and new users.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:22Motivation
- 1:57Data Collection
- 3:38Experiment Setup
- 3:59Word-level Performance
- 4:49Sentence-level Performance
- 5:20System Performance
- 5:34Conclusion
related talks
01h 04m“Hello, AI?!” – Real-Time Voice Interactions with LLMs
48mConversational ULS: Talking to Siri, Alexa, and Your Web Browser
19mHow Captions Enables Anyone To Tell Their Story With Video
28mIndirect Prompt Injection Into LLMs Using Images and Sounds
06mVocalLock: Sensing Vocal Tract for Passphrase-Independent User Authentication Leveraging ...
16mEchoLLM: LLM-Augmented Acoustic Eavesdropping Attack on Bone Conduction...
39mLive voices with LiveView
14mBeyond Tokenization - Shaping the Future of Conversational AI