Machine Learning in Fastly's Compute@Edge
Talkat WasmCon 2023 · on Wednesday, 2023-09-06· at 15:45· Grand Ballroom I, Hyatt Regency Bellevue, 900 Bellevue Way NE·
summary
Fastly’s Compute@Edge runs ML by extending wasi‑nn so models load once named models before module instantiation, avoiding per‑request reloads and enabling cached, low‑latency inference; a MobileNetV2 image demo returns ~96% for a killer whale and a Stable Diffusion pipeline runs in about 5–6 seconds via a new KServe backend that proxies to OpenVINO, ONNX and PyTorch backends. Host APIs become async in Wasmtime to avoid event‑loop stalls, and ML backends move out‑of‑process for process isolation to contain malicious models; Wasm module size is limited (~128 MB) while models may be GBs.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:23Generative AI transforming knowledge work
- 3:14Waz DNN for ML in WebAssembly
- 5:44Waz DNN history and evolution
- 7:52Function as a service with ML at Fastly
- 8:56Security challenges with ML models
- 10:22Compute@Edge architecture and ML
- 12:48Decoupling model lifecycle from requests
- 14:35Protecting environment from malicious models
- 17:26Running models where best
- 19:33K-Serve backend for ML inference
- 20:15Live demo: ML inference at edge
- 24:37Q&A
related talks
33mWebAssembly based AI as a Service with Kubernetes
36mApplied AI on the edge
37mDeep Learning Neural Network Acceleration at the Edge
22mMachine Learning with Wasm (wasi-nn)
42mAI Everywhere– accelerate your development from PC to cloud
33mMachine Learning in the Browser: Fast Iteration with ONNX & WebAssembly
10mDeploying machine learning in wasm to CPU, GPU and NPU with the new wa...
53mMachine Learning Platforms