WebLLM: A high-performance in-browser LLM Inference engine
summary
WebLLM is a browser-native LLM inference engine that runs models 100% client-side, compiles WebGPU kernels via MLC-LLM/TVM, and uses WebAssembly for CPU/runtime tasks. It exposes an OpenAI-API style interface and a ServiceWorkerMLCEngine for background GPU execution, enforces structured JSON outputs via XGrammar, and supports embeddings and VLMs (Phi-3.5). Measured results show about 85% of native performance and ~30–70 tokens/sec on laptops for 3B 4-bit models.
AI-written summary · about
chapters (8)
- 0:00Introduction to local AI
- 2:19Benefits of local LLMs
- 3:20Browser as a platform
- 5:38WebLLM design principles
- 7:51API features and usability
- 11:23WebGPU acceleration
- 13:55Browser runtime adaptation
- 15:57Future work and conclusion
related talks
31mFrom ML to LLM on device AI in the browser
34mPrivacy First: Building LLM-Powered Web Apps with client side WASM
17mRunning WebLLM in the Browser on RISC-V Toward Lightweight, Local AI Experiences
20mFrom ML to LLM: On-Device AI in the Browser
46mFrom ML to LLM: on-device AI in the browser
55mFast And Portable LLM Inference With WebAssembly And Rust
30mP2P Distributed LLM Inference Across Browsers With llama.cpp And WebRTC
40mOpen Source LLMs in the Cloud: Scalable Solutions