Apple Vision Pro New Interface Based on Hand Tracking

on YouTube ↗23m

" summary

A Swift/SwiftUI implementation recreates visionOS SystemOverlay behavior on Apple Vision Pro using SpecialTrackingSession hand tracking, RealityView attachments and a BillboardComponent, with Combine plumbing to detect flips and taps. The per-frame (~90Hz) logic treats finger distance under a 3 cm threshold as a tap and reliably opens app windows, but it cannot access gaze (privacy-locked), may be overridden by the OS persistent system overlay, and currently handles only the right hand.

AI-written summary · about

" chapters (14)

1. 0:00Introduction
2. 2:10Spatial computing interface
3. 4:36System overlay in visionOS 2.0
4. 6:01Custom system overlay demo
5. 8:10Hand tracking with visionOS 2.0
6. 11:01Hand gesture system
7. 13:50Overlay between fingers
8. 14:10Hand flips for view switching
9. 15:35Finger tap gestures
10. 17:19Opening windows with gestures
11. 18:03Gaze tracking challenges
12. 19:47Persistent overlay conflicts
13. 21:13Tracking session limits
14. 21:55Future of visionOS

" related talks

• 38mBuilding Responsive InteractionsAdam Bell · iOS Conf SG 2023
• 37mGetting Focus Right on Apple PlatformsPradnya Nikam · iOS Conf SG 2024
• 30mEvolving Existing Projects With SwiftUIPeter Steinberger · Swift Heroes Digital 2021
• 35mSwiftUI Performance for Demanding AppsAviel Gross · SwiftLeeds 2023
• 41mMulti Platform Development, the Swiftui wayMalin Sundberg · 360|iDev 2021
• 27mReusable UI components with SwiftUIMira Kim · /dev/world 2019
• 06mMaking Portrait mode yourselfRina Kotake · try! Swift Tokyo 2019
• 18mSwift as LightJon-Tait Beason · try! Swift Tokyo 2019