UniKey: Enabling Surface-Based Typing with Commodity Smartwatches via Cross-Modal Learning
summary
UniKey uses only a commodity smartwatch—smartwatch microphone for inter-keystroke timing and an IMU-based left/right hand signal—to map tap timing and handedness to words. It pretrains on a user’s regular physical-keyboard timing (no smartwatch training), requires three calibration taps per hand, then decodes candidates with a classifier plus LLM re-ranking. Tested with 10 typists and over 2,700 minutes of typing, it achieves roughly 6.45% top‑5 WER (about 6.5% retry, 30% selection rates) on hard stable surfaces.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 0:28Motivation and challenges
- 1:55UniKey system overview
- 4:01Keystroke detection pipeline
- 5:24Word decoding with LLM
- 6:46Evaluation and results
- 8:50Limitations and improvements
- 10:13Conclusion and future work
related talks
21mThe Impact of Word, Multiple Word, and Sentence Input on Virtual Keyboard Decoding Performance
19mATK: Enabling Ten-Finger Freehand Typing in Air Based on 3D Hand Tracking Data
06mWrite, Attend and Spell: Streaming End-to-end Free-style Handwriting Recognition Using Smartwatches
14mLeveraging Dual-Observable Input for Fine-Grained Thumb Interaction Using Forearm EMG
03mInterview with Ben Medlock
06mDesigning and Evaluating Hand-to-Hand Gestures with Dual Commodity Wrist-Worn Devices
24mSecurity at your fingertips: Theory → Practice!
20mModelling Learning of New Keyboard Layouts