How Foundation Models can transform interactions on visionOS
summary
Apple’s on-device Foundation Models power a Vision Pro escape-room prototype called Green Spurt that turns scattered lore and FAQs into an immediately queryable glossary without opening menus, using on-device Foundation Models for privacy and low latency. A chatbot companion "Splat" handles messy user phrasing and detects intent; attempts to auto-generate puzzle solutions fail, so the shipped design uses a hybrid AI + deterministic hints system where AI flags need and deterministic steps reveal hints, improving presence and playability.
AI-written summary · about
chapters (19)
- 0:00Foundation models on visionOS
- 1:03Journey to spatial computing
- 3:31AI landscape shifts
- 7:20Why foundation models matter
- 8:03Navigating info with AI
- 9:50Natural language actions
- 10:32Personalized spatial AI
- 11:16AI in Green Spur game
- 13:44Splat companion demo
- 16:34Handling messy queries
- 17:16AI limits for hints
- 19:21AI + deterministic systems
- 20:46AI for presence support
- 21:53Foundation models examples
- 23:39Natural language customization
- 24:24AI model limitations
- 24:45Experiment with interactions
- 25:48Redesigning tech relationships
- 26:10Q&A
related talks
13mGuided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models
56mBeyond Chat: Bringing Models to the Canvas
01h 04mBuilding Expertise Into Generated Conversations
21mBeyond the Chatbox: What Changes When AI Runs Where You Are
59mThe Evolution of the Player Interface.
32mBeyond Vibe Coding: Leveraging Foundation Models to Infer Intent
25mAI-Assisted Swift in Gamifying an iOS App
38mCreate the next generation of user experiences with text generation