MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving
Talkat USENIX ATC '19 · on Friday, 2019-07-12· at 12:30· USENIX ATC Track II: Grand Ballroom VII–IX·
summary
MArk is an AWS inference-serving system that combines dynamic batching (to make GPUs cheaper than CPUs at moderate batch sizes), predictive autoscaling (to hide VM launch latency) and serverless fallbacks (Lambda) plus spot instances to handle spikes and discounts. Implemented across TensorFlow/MXNet/Keras backends, it runs proactive provisioning and greedy instance packing, achieves up to 7.8× cost reduction versus SageMaker while improving tail latency and meeting SLOs (99th-percentile targets under one second).
AI-written summary · about
chapters (8)
- 0:00Introduction to Mark
- 1:44Challenges in ML serving
- 3:36Hardware and performance
- 4:50Cloud service trade-offs
- 7:18System design insights
- 9:43Mark system architecture
- 12:20Performance evaluation
- 15:48Conclusion and Q&A
related talks
41mDeploy ML models for inference at high performance & low cost
34mBuilding scalable end-to-end deep learning pipelines in the cloud
28mScaling Machine Learning Training and Inference
12mLeveraging AWS and Kubernetes for Scalable AI/ML Workloads in the Cloud
32mOptimizing AI/ML workloads for sustainability
35mMLDS 2019: Madhusudan Shekar of Amazon at Machine Learning Developers Summit 2019
43mDesigning and Building Complex Machine Learning Projects
44mPragmatic Machine Learning in the Cloud