Open Source LLMs in the Cloud: Scalable Solutions
summary
Three deployment paths: Python stacks (LangChain, LlamaIndex), native runtimes (llama.cpp, vLLM) and WebAssembly via WasmEdge and WASI NN; claim is that 16–20 MB WASM images plus tiny model plugins (Whisper ~3 MB) let operators mount models outside containers and swap backends without recompiling. A working demo wires Whisper + Llama 3.2 to translate video English→Japanese, and a decentralised node project reports >200k nodes and high token throughput.
AI-written summary · about
chapters (8)
- 0:00WasmMatch and WebAssembly intro
- 4:34Scaling LLM cloud challenges
- 9:15LLM deployment approaches
- 14:27Wasm container tech details
- 23:21Wasm container dependencies
- 28:47Live demo: Wasm AI services
- 33:54Summary and open-source
- 35:44Q&A: dependencies & upgrades
related talks
55mFast And Portable LLM Inference With WebAssembly And Rust
33mEfficient and Cross-Platform LLM Inference in the Heterogenous Cloud
45mEverything You Need to Know About Running LLMs Locally
30mP2P Distributed LLM Inference Across Browsers With llama.cpp And WebRTC
27mWasm Is Becoming the Runtime for LLMs
34mPrivacy First: Building LLM-Powered Web Apps with client side WASM
16mReact + WebLLM: Building Intelligent Applications with In-Browser Language Models
48mWorkshop: Efficient and Portable AI / LLM Inference on the Edge Cloud