Look ma, no hands! Multimodal AI-Agents in the browser
summary
Demonstration shows a full multimodal agent running entirely in the browser with transforms.js and WebGPU/WebAssembly, no cloud APIs or install required. It uses Whisper tiny for ASR, Kakoro for TTS, vision language embeddings, and MCP for decoupled tool access. Small LLMs like Gemma 3 (270M) and Quen 3 run on device at ~52 tokens/sec (200 tokens in ~3.7s). Voice activity detection processes 113 chunks of 32ms at ~1.3ms per chunk. The Jarvis prototype works end to end.
AI-written summary · about
chapters (16)
- 0:00Multimodal AI agents intro
- 1:07Jarvis-like AI assistants in JS
- 2:53ML models in browser with Transformers.js
- 5:24AI agents and language models
- 6:49Comparing large, small, and open-source LLMs
- 9:16Tiny LLMs in browser
- 11:06Tool calling for AI agents
- 13:53Structured outputs for tool calls
- 14:16Model Context Protocol
- 15:40Audio input/output for AI agents
- 16:22Whisper speech transcription
- 17:47Voice activity detection
- 19:53Fully multimodal Jarvis
- 21:38Custom MCP servers
- 23:47Vision language models in Jarvis
- 27:29Key takeaways
related talks
29mClient side Web AI Agents Building smarter user experiences for a future age
22mBoring but Effective: WebAI That Ships
43mLet’s build K.I.T.T. with JavaScript
02mDiscover Web AI: Client side Agents, Gen AI, and machine learning in the browser
46mFrom ML to LLM: on-device AI in the browser
23mWebAI: Building the next user experience in the browser
20mFrom ML to LLM: On-Device AI in the Browser
53mAI Didn’t Kill the Web, It Moved in!