The RAod to intent-driven automation
summary
Demonstrates a working JARVIS‑style system that automates UIs by translating intents into mouse/keyboard actions and evaluative checks, successfully generating a poem in the demo. Combines large language models, multimodal vision models (OpenAI, Llama 3.2), and Vision agents that produce structured JSON which a controller executes via PyAutoGUI; bounding‑box errors from general vision models are fixed by grounding with specialized detectors. Code for the intent pilot is open source.
AI-written summary · about
related talks
43mThe Road to Intent-Driven Automation- 43mThe Road to Intent-Driven Automation
29mGuiding Agentic AI with Vue and Pinia
29mPairing with the machine: How AI Agents are rewriting the rules of software development
35mBuilding AI-powered user interfaces
31mBuilding and Orchestrating Production-ready Agentic AI Systems
44mAI Driven UI Automation, API Automation and CICD Integration
30mLook ma, no hands! Multimodal AI-Agents in the browser