Preview: ML Serving
summary
Three systems target cloud-scale DNN inference with concrete claims and prototypes: PET compiles tensor programs using partially equivalent transformations plus an automatic correction pass to explore a larger optimization space and reduce execution time (with some measurable accuracy recovery); Inf-as-a-Service runs a runtime scheduler and autoscaler component that chooses backend, model variant (batch-1, batch-8, batch-16) and prioritization to meet per-request SLOs (e.g., 100 ms) across heterogeneous users; Paleon detects class skew in video streams (ImageNet: ~1M images, 1000 classes) and swaps compact models to cut mobile power.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:28Machine Learning as a Service
- 2:30Inference Serving at the Cloud Scale is Difficult
- 4:21Overview Papers at ATC 21 and OSDI 21
- 5:17PET - Optimizing Tensor Programs Partially Equivalent Transformations and Automated Corrections
- 7:29INFaas Automated Model-less Inference Serving
- 9:04Palleon Runtime System for Efficient Video Processing
- 11:24Program Wednesday, July 14
related talks
36mAccelerating High-Performance Machine Learning at Scale i...
36mAccelerating Machine Learning at Scale with
30mAccelerating AI Application: NAVER's Journey from Model to Application
36mApplied AI on the edge
19mThe Small Model Infrastructure Nobody Built (So We Did)
41mEscaping the Cloud: High-Performance AI in your Browser
41mInfrastructure for ML Applications
27mModel Serving at the Edge Made Easier