Multi Runtime Serving Pipelines
summary
An open source runtime treats models as serverless functions with an Envoy sidecar handling API, discovery and routing; pipelines chain heterogeneous runtimes (fastText, TensorFlow, JVM/C++ libs) and run a demo SSD detector split 50/50 GPU/CPU split with GPU ~300 ms and CPU ~6 s latencies. The system emits per-stage metrics and Zipkin traces for latency and drift detection, rejects PMML-only export because of duplication, limited extensibility and extra moving parts.
AI-written summary · about
chapters (12)
- 0:00Introduction to talk
- 1:24Model deployment in production
- 3:32Operational requirements for models
- 6:49Monitoring model quality online
- 8:58Multi-runtime serving architecture
- 10:03Demo with SSD model
- 12:11Building pipelines from models
- 14:02Monitoring pipeline metrics
- 15:51Why not single neural networks
- 17:17Need for modular pipelines
- 17:57Why not TensorFlow Serving
- 20:04Q&A
related talks
36mAccelerating High-Performance Machine Learning at Scale i...
22mCustom Pipeline Parallelism for Frontier-Scale Model Serving
23mHeterogeneous Pipelines as the Unlock for ML at Scale
33mDeploying and Scaling Spark ML and Tensorflow AI Models
21mOptimizing Model Serving on Kubernetes With Model Streaming
40mBuilding highload ML powered service
12mPreview: ML Serving
16mRevisiting Pipeline Parallelism for LLM Serving