Privacy First: Building LLM-Powered Web Apps with client side WASM
summary
Demonstrates a complete on‑device LLM pipeline: quantize and compile models to run in‑browser, ingest PDFs, embed text into a local vectorstore, and perform retrieval‑augmented generation with an LLM like Ollama and tooling such as Transformers.js and Voy. Shows a Whisper transcription running as a ~1 GB download, an example that indexed 220 entries into IndexedDB and answers queries offline, and explains use of WebAssembly+WebGPU and MLC for performance plus the accuracy/performance tradeoffs and occasional crashes.
AI-written summary · about
chapters (15)
- 0:00Privacy-first LLM web apps
- 4:18Privacy risks of cloud AI
- 6:02Cost of cloud AI APIs
- 7:07WebGPU and WASM for AI
- 10:16Auto-selecting device backends
- 13:31TensorFlow.js and Transformers.js
- 15:37Live Whisper transcription
- 18:27LLMs in the browser
- 21:59Quantizing models for browsers
- 23:03WebLLM for on-device inference
- 24:30Retrieval augmented generation demo
- 28:44Chrome’s Gemini Nano
- 30:32Hardware and future of on-device AI
- 31:34WASM for production AI?
- 33:02Conclusion and Q&A
related talks
20mFrom ML to LLM: On-Device AI in the Browser
17mWebLLM: A high-performance in-browser LLM Inference engine
31mFrom ML to LLM on device AI in the browser
46mFrom ML to LLM: on-device AI in the browser
17mRunning WebLLM in the Browser on RISC-V Toward Lightweight, Local AI Experiences
16mReact + WebLLM: Building Intelligent Applications with In-Browser Language Models
40mOpen Source LLMs in the Cloud: Scalable Solutions
30mP2P Distributed LLM Inference Across Browsers With llama.cpp And WebRTC