Building model testing infrastructure and scaling AI predictions
summary
Describes a working serverless MapReduce using S3 and AWS Lambda with a coordinator Lambda that processes 4,200 files (424 GB, 3.54 billion lines) in about 2 minutes for $0.53. Shows Lambda-hosted inference for ResNet-18 and SqueezeNet (latencies ~1.2–1.5 s at 75 rps, ~380 ms at 100 rps), argues inference drives ~95% of cost, stores models in S3 and caches in memory, and deploys a multi-armed bandit routing design using DynamoDB/S3 for metric collection, versioning and gradual rollouts.
AI-written summary · about
chapters (13)
- 0:00Introduction and background
- 1:15Machine learning pipeline overview
- 3:25Serverless MapReduce paradigm
- 6:35VPC flow logs demo
- 12:24K-means clustering with MapReduce
- 16:16Deep learning phases and inference cost
- 19:24Deploying models on Lambda
- 23:16Geolocation model demo
- 25:05Model performance benchmarks
- 27:58Object detection with Lambda
- 29:24Model deployment challenges
- 31:31Multi-armed bandit for model evaluation
- 35:46Serverless model deployment strategy
related talks
32mServerless Data Processing and Machine Learning
34mBuilding scalable end-to-end deep learning pipelines in the cloud
18mServerless AI's Rise: Revolutionizing ML Deployment
12mServerless Deep Learning
46mServerless for Data Scientists
01h 00mServerless - reality or BS - notes from the trenches
41mBuilding Serverless Machine Learning Models in the Cloud
48mBuilding Serverless AI powered Applications on AWS