Scalable machine-learned model serving
summary
Shows an architecture that pushes computation to data shards, copying TensorFlow and ONNX models to C++ content nodes so each node evaluates models locally in parallel. Benchmarks claim near linear scaling with partitions versus TensorFlow Serving hitting a bandwidth bottleneck. Demonstrated on Vespa.ai running hundreds of production applications (Verizon Media/Yahoo), serving ~250,000 queries/sec and evaluating models over millions of data items per request, with sub-100ms latencies.
AI-written summary · about
chapters (12)
- 0:00Introduction to big data serving
- 1:56Big data maturity levels
- 7:37Vespa’s origins in web search
- 11:51Use cases for Vespa
- 15:30Vespa vs analytics systems
- 17:00Vespa architecture overview
- 20:38Low-latency computation strategies
- 25:15Data distribution in Vespa
- 27:53Machine learning inference engine
- 31:07Releases and open-source process
- 32:09Summary and getting started
- 37:19Q&A session
related talks
40mServing Machine Learning Models at Scale Using KServing
36mAccelerating Machine Learning at Scale with
39mIntroducing KFServing: Serverless Model Serving on Kubernetes
31mServing Machine Learning Models at Scale Using KServe
36mAccelerating High-Performance Machine Learning at Scale i...
41mBig Data Serving: The Last Frontier. Processing and Inference at Scale in Real Time
34mServing ML Models at Scale with Seldon and Kubeflow
31mFaster Model Serving with Ray and Anyscale