Develop with Purpose – Building Sign Language Recognition for VisionPro
summary
Builds a proof of concept ASL recognizer for VisionPro that uses Vision Pro hand tracking provider and world tracking provider because raw camera frames are unavailable to non‑enterprise apps, extracts fingertip knuckle wrist XYZ per frame, trains a tabular Random Forest in Create ML for static letters then an activity classifier with a 50 fps sliding window (100 samples, ~2s) for sequences, achieves high accuracy on most letters but fails on motion letters J and Z, and requires much larger, multiuser data plus facial expression input to be production ready.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:05Computing paradigm shifts and Vision Pro
- 4:12AR projects and Vision Pro inspiration
- 5:38Accessibility purpose for VisionOS
- 6:21VisionOS APIs for sign recognition
- 7:22ML model training challenges
- 8:49Hand tracking data for ASL
- 10:58Training letter model with Create ML
- 12:25Full alphabet results
- 13:09Accuracy with gestures and sequences
- 14:32Two-hand positions and word prediction
- 16:43Classifier integration
- 18:15Results limits and future work
- 19:19Open source and accessibility goals
- 19:41Conclusion
related talks
06mTeaching American Sign Language in Mixed Reality
44mIntroduction to Apple Vision Pro development
58mAccessible Spatial Computing
14mTransform intent into action through your eyes
41mComputer Vision for Remote AR
38mAI + AR Superpowers!
54mMe, Myself and eyeOS
13mVRSight: An AI-Driven Scene Description System to Improve Virtual Reality Accessibility for Blind...