Creating General User Models from Computer Use
summary
A system-level architecture builds a general user model from screenshots and other unstructured interactions, converting observations into confidence-weighted natural language propositions and updating them via an observe propose retrieve revise loop. Implementations run on open-source models and an internal infra to augment chat assistants, manage OS notifications, and power proactive assistants called GUMBOs that discover and execute suggestions. Evaluation: 18 participants on email inboxes and a 5-participant, 5-day study; >90%‑confidence inferences are nearly perfect, 80% → 88.3%, 60% → 70%, 40% → 60%; removing loop components degrades performance.
AI-written summary · about
chapters (9)
- 0:00Introduction and award
- 0:26What is a general user model
- 3:56How the model works
- 5:40Applications for privacy
- 7:03Mixed initiative interaction
- 8:24Evaluating accuracy and calibration
- 9:52Proactive assistant demo
- 10:52Future implications
- 11:13Q&A session
related talks
01h 04mBuilding Expertise Into Generated Conversations
47mOperationalization of AI/ML models and use cases for ChatGPT Q&A
34mWhen Software Understands the User
48mAI Supercharging Product Management: Go Beyond Yourself!
48mMastering the AI Shift: Adaptive Strategies for Software Evolution
35mThe New User Interaction Model
01h 04mBuilding Expertise Into Generated Conversations
23mUser Experience Now Includes Machine Experience