Serverless Data Processing and Machine Learning
summary
A MapReduce-style pipeline uses Amazon S3 as shared storage and AWS Lambda for mappers, reducers and a coordinator, processing 420 GB (3.5 billion lines, 4,200 files) in under 200 seconds for about $0.52 total. K‑means is implemented atop that framework; image inference (SqueezeNet/ResNet) runs in Lambda with median latencies around ~1.1 s depending on memory sizing, trading cost for speed. Notes include the 15‑minute Lambda execution limit, numpywren/Spark‑on‑Lambda experiments, and deployment via AWS SAM plus alias‑based canary/multi‑armed rollout.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:22Serverless application
- 6:38X-Ray Service Map
- 7:05Architectural Patterns
- 9:35Serverless MapReduce Architecture
- 11:15Demo: VPC Flow Log Processing
- 16:22Neural Networks and AWS Lambda
- 18:55ImageNet models on Lambda
- 19:57Benchmarks
- 23:02Other Open Source "HPC like" Projects
- 24:26The Pattern
- 26:17Beyond traditional CI/CD Pipelines
- 27:34Multi Arm Bandit
- 28:26MultiArm Bandit for Lambda Inference
- 29:55Take Home Messages
related talks
39mBuilding model testing infrastructure and scaling AI predictions
41mUnlocking Faster & Efficient Data Processing w/ Serverless
21mThe best compute for your storage - Amazon S3 & AWS Lambda
46mServerless for Data Scientists
34mDesigning Data Intensive Applications In Serverless Architecture
34mBuilding scalable end-to-end deep learning pipelines in the cloud
47mServerless in production, an experience report
19mHow to create a fast and cost effective serverless geospatial analysis pipeline...