Understanding the Mathematical Foundations of Spatial Computing
summary
Demo shows an open‑source gesture‑matching framework that uses MediaPipe on macOS to stream hand keypoints into the Vision Pro simulator (available on GitHub and as an SPM package). Matching uses cosine similarity (dot product) on per‑joint position vectors to score gestures (examples jump from ~40 to ~95 points when matching an OK sign). Implementation also uses cross products to compute beam direction, mixed product for UV/volume tricks, and 4×4 column‑major matrices with inverse/mirroring to handle left/right hands.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 2:40Demo gesture matching framework
- 6:393D coordinate systems
- 7:45Points vs vectors
- 10:12Vector operations
- 17:07Matrix basics
- 27:19Matrix operations
- 34:33Summary of concepts
- 35:53Vision Pro development demo
- 37:28Q&A session
related talks
20mToward Realistic Hands Gesture Interface: Keeping it Simple for Developers and Machines
21mPosePIE: Replace Your Keyboard and Mouse With AI-Driven Gesture Control
28mBuilding Motion Controlled Websites in JavaScript
27mSign Language Recognition Improvements Since the London Keynote
20mPathSync: Multi-User Gestural Interaction with Touchless Rhythmic Path Mimicry
33mSee More, Be More
11mEfficient Finger Model and Accurate Tracking for Hover-and-Touch, Mid-air and Microgesture Intera...
02mRecap Video