INFaaS: Automated Model-less Inference Serving
summary
INFaaS automatically generates and profiles hundreds of model variants from trained models, then picks per-query combinations of model, framework, and hardware and autoscaling actions to meet latency and accuracy targets while minimizing cost. It implements a model-less API, VM-level and per-worker autoscaling including a novel model vertical scaling (upgrade/downgrade between variants), uses a heuristic over an integer-program objective, and reports 1.3× higher throughput and up to 21.6× cost savings on AWS EC2.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:18Model Registration
- 1:40Diverse application requirements
- 2:27Today's Inference serving
- 3:45Goals & Requirements
- 4:11INFaaS overview
- 5:38The model-less abstraction
- 7:57Existing systems
- 8:35Autoscaling
- 9:13Model-Autoscaler at each worker
- 11:35Evaluation
- 12:31How well does INFaaS scale with load?
- 13:24Putting it all together
- 14:19Conclusion
related talks
35mAdvanced Model Inferencing Leveraging KNative, Istio & Kubeflow Serving
20mMArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving
28mScaling Machine Learning Training and Inference
15mAI Model Inference on K8s
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
27mAI as a Service Scaling AI with Ease and Innovation
12mAI-Powered Predictive Analytics for Multi-Cloud Management
24mInference on Streaming Data at Scale at Intuit