Deploying computer vision models at scale devops-free
summary
Veriff moves from one-model-per-microservice to serving many models from a shared GPU pool using Triton Inference Server packaged into model repos and stored on S3, fronted by an AWS Multi-Model Endpoint that loads models on demand and autos-scales instances. CI builds Triton artifacts from a monorepo, clients emit Prometheus model-level metrics, and Triton features like dynamic batching raise throughput. Result: deployment time drops to one day and inference costs fall ~75%, with retries needed to handle autoscaling cold-load timeouts.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:11Company use case
- 3:21Microservice challenges
- 6:54Triton Inference Server
- 10:06AWS SageMaker MME
- 13:56Implementation overview
- 17:49Results and benefits
- 18:57Limitations and future
- 20:22Q&A
related talks
25mMoving Inference to Triton Servers
32mScaling Inference Deployments with NVIDIA Triton Inference Server and Ray Serve
51mNvidia Triton Inference Server: building production ML without developers
32mHow Cookpad Leverages Triton Inference Server To Boost Their Model S...
51mSpeeding up deployment of machine learning models
54mDeep learning on AWS with NVIDIA: From training to deployment
31mFaster Model Serving with Ray and Anyscale
46mServing machine learning models as an inference API in production