From perception to action: World Foundation Models (WFMs) for autonomous robots and vehicles
summary
World Foundation Models replace modular stacks with an end-to-end AI that turns multi-camera and lidar inputs into control trajectories by training on synthetic, geo‑specific video and point‑cloud data. Implementation uses a diffusion backbone constrained by 3D bounding boxes and map layouts plus depth/segmentation to produce six-view outputs that match simulator scenarios (CARLA) and Indian road features like potholes, buses and overpasses. Models generate ~100–500 frame clips, cut data collection needs, but struggle with tiny traffic signs, lidar fidelity and compute cost.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 0:22Conventional autonomy limits
- 3:12Long-tail problem breaks systems
- 5:42Feature overload acronyms
- 7:07End-to-end AI models
- 9:13Data needs for training
- 11:00World Foundation Models
- 13:08WFMs generate scenarios
- 14:54Geo-specific data generation
- 16:41Synthetic data for impossible real-world scenarios
- 17:43Scenario validation for testing
- 18:25Quality metrics for driving data
- 19:33End-to-end validation metrics
- 21:58Data flywheel for improvement
- 23:41Autonomy 2.0 benefits
- 24:23WFMs open challenges
- 26:28Q&A
related talks
08mUnified Perception: Models for Autonomous Driving
27mDriving Autonomous Vehicles Forward w/Real World Applications
21mBuilding a New Foundation Model
49mUsing Deep Learning for Perception in Autonomous Systems: A Perspective from Control Theory
31mDeep learning for self-driving vehicle technology
42mHow We Created a Thinking Robot?
41mAI applications in autonomous driving
24mContext Aware Software Stacks for Mobility: Composive.ai Overview