From ML to LLM on device AI in the browser
summary
Shows running TensorFlow.js models in-browser with a WebGPU backend: face landmark demo jumps from 6 fps (CPU) to 12 fps (WebGL) to ~32 fps (WebGPU). Trains a custom Swiss‑German wakeword with Teachable Machine (≈50–70 samples) and wires it into a slides extension buzzer. Compiles LLMs with Apache TVM and WebLLM to run Gemma 2B locally after quantization to 4‑bit, achieving ~130 tokens/s encode and ~36 tokens/s decode on an M3 laptop; Windows i9 drops to ~4 tokens/s. Also demos fast on‑device RAG: PDF parse 170 ms, vectorize ~3 s, retrieve 32 ms, generate ~4.2 s.
AI-written summary · about
chapters (9)
- 0:00Introduction and motivation
- 2:40Browser-based computer vision
- 7:18Hand tracking and input
- 9:36Audio and keyword recognition
- 12:00Running LLMs in the browser
- 16:57Practical AI applications
- 20:23RAG for browser-based AI
- 24:37Future of web AI
- 29:56Closing thoughts
related talks
20mFrom ML to LLM: On-Device AI in the Browser
46mFrom ML to LLM: on-device AI in the browser
17mWebLLM: A high-performance in-browser LLM Inference engine
34mPrivacy First: Building LLM-Powered Web Apps with client side WASM
16mReact + WebLLM: Building Intelligent Applications with In-Browser Language Models
39mWebAssembly-Powered Machine Learning
47mOwn Your LLM: Running Generative and Agentic AI Locally
08mDemo: Gemma on-device with MediaPipe and TensorFlow Lite