On scaling Machine Learning microservices
summary
A checklist of concrete tactics: a prototype built in hours survives production with thousands of users per day by using concurrency (async or ThreadPoolExecutor) for IO, caching via functools.lru_cache or TTL caches, and FastAPI background jobs to return job IDs immediately; package apps with small multi-stage Docker images (use Dive to find bloat) and choose FaaS/PaaS/ECS based on control vs startup time; pressure-test with Locust to find the 95th-percentile choke point; expose a batch predict endpoint and use numpy/pandas for C-backed speed.
AI-written summary · about
chapters (11)
- 0:00Intro
- 1:50Prototype
- 3:26Productionizing
- 16:17Serving
- 17:12Batch Predict
- 18:47Handling the load
- 19:33Farts in the wind
- 20:20Locust
- 22:25BCG
- 22:38Summary
- 23:37Questions
related talks
21mHarnessing Microservices for Scalable, Fault-Tolerant ML Systems
14mPerformance Optimization in High-Volume Transaction Systems
26mMLOps for the rest of us- A poor man's guide to putting models in production
26mCreating a scalable ecosystem of Microservices
13mMLOps: From AI Research to Production-Ready Systems
35mA simple way to validate and monitor the performance of your ML Applications.
28mProductionizing Machine Learning Models at Scale with Kubernetes
37mHow the hell do I run my microservices in production, and will it scale