Scaling Machine Learning Training and Inference
summary
Vedrai runs thousands of forecasting models with an AWS-driven training pipeline that is 100% serverless training using AWS Step Functions + Lambda + AWS Batch (Fargate) to dispatch and parallelize per-store time series, tracking experiments in Databricks MLflow + Delta Lake and packaging models as Docker containers. Real‑time serving uses Databricks endpoints with a ~3,000 QPS limit, token‑bucket throttling proves too slow, and heavy simulation loads push the team to a streaming solution via Kafka / Amazon MSK, trading lower latency for constant cost.
AI-written summary · about
chapters (7)
- 0:00Introduction and background
- 2:55Product overview and challenges
- 5:21Serverless training infrastructure
- 10:29Real-time inference scaling issues
- 12:20MLflow model training demo
- 18:05Model deployment strategies
- 22:40Q&A session
related talks
34mBuilding scalable end-to-end deep learning pipelines in the cloud
34mDeploying a Machine Learning System That Only Runs Locally
29mMachine learning in production
19mDrive transformation through machine learning
20mMArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving
39mBuilding model testing infrastructure and scaling AI predictions
27mFrom Training to Serving: Machine Learning Models with Terraform
20mIndustrial-Scale Machine Learning