We Thought the Digital World Would Be Easy to Simulate. Was it?
summary
Argues that simulating the digital world is hard in practice but solvable by models that simulate language-form interfaces rather than pixels. Uses VLMs embodied with tools (screenshots, mouse/keyboard, browser APIs) trained on data collected from thousands of Docker containers and virtual machines and on software docs, video tutorials and crowdsourced workflows. Reports movement from ~20–30% to 80–90% success on 400 OSWorld tasks within a year, trains with supervised fine-tuning plus RL, and shows a 35B model matches frontier performance* while running a 3D CAD logo demo successfully.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 6:34Training VLMs for computer use
- 10:07Progress in computer use
- 12:36Challenges in simulation
- 17:30Training methods
- 19:56Specialized models
- 20:17Demo and closing
related talks
19mThe Physical Turing Test: Solving General Purpose Robotics
44mThe LLM Revolution: From Inception to the Future of Thinking Machines
40mSurviving the Agentic Hype with Small Language Models
22mBoring but Effective: WebAI That Ships
33mOpening Keynote: LLMs, between fantasy and reality - the poin...
30mYou are not a Computer: Python, Dynamical Systems, and Radical Embodied...
01h 19mBuilding an LLM on your Laptop to...
17mCode World Model: Building World Models for Computation