Serving machine learning models as an inference API in production
summary
Deploys a PyTorch cell‑detection model as a JSON inference API using model weights on S3 and TorchScript tracing; an AWS Lambda + layer + temp‑extraction hack works but hits 250MB limits, no GPU, cold starts ~4s and API Gateway timeouts, so it proves too slow for 1000×1000 images. Switching to Docker + GPUs (nginx → gunicorn → Flask → traced model) yields first‑call calibration delays (~9s) then <1s responses and scales with Docker Swarm across GPU servers; metrics go to Postgres and Grafana; cloud GPUs off‑AWS cut costs ~7×.
AI-written summary · about
chapters (14)
- 0:00Start
- 2:29Cancer diagnostics today
- 3:59Cancer diagnostics tomorrow
- 4:46Let's do it!
- 6:17Example: cancer cell detection
- 7:49Goal: API Integration of the model
- 8:08First option: serverless lambda function
- 10:03Disadvantages Lambda Function
- 11:41Cool possibility: lambda layers
- 12:51Trick: lambda layer /tmp extraction
- 13:34Live Demo Serverless Tumor Cell Detection
- 18:22Second option: GPU servers with Docker
- 19:44Disadvantages Docker
- 34:50We're hiring devs and data scientists!
related talks
12mServerless Deep Learning
51mSpeeding up deployment of machine learning models
39mBuilding model testing infrastructure and scaling AI predictions
15mServerless Tensorflow 1.0.0 on AWS. Wait what?
41mBuilding Serverless Machine Learning Models in the Cloud
40mHow to Deploy Machine Learning Inference Code to Production
37mFrom Hours To Milliseconds: Scaling AI Inference 10x With Serverless on Kubernetes
43mProblems and Solutions with AI Model Deployments in k8s