Building Developer-Centric ML Inference Platforms
summary
Platform claims handle 300–400+ models at scale using Kubernetes CRDs and operators, Databricks for training and a feature store fed by Kafka streams for online/overnight features. Deployment uses canary/gradual rollouts (1%, 5%, 10%) with automated validation and rollback, autoscaling on GPU/TPU, request batching and multi-level cache to cut latency. Monitoring detects feature and prediction drift, enforces PII masking, GDPR audits and artifact versioning for fast rollbacks.
AI-written summary · about
chapters (13)
- 0:00Introduction to Developer-Centric ML Inference Platforms
- 0:10Challenges in Building ML Inference Platforms
- 2:50Architectural Foundations for ML Platforms
- 6:05Kubernetes and CRDs in ML Workflows
- 8:54Automated CI/CD for ML
- 11:19Feature Store and Data Pipeline Architecture
- 12:13Model Serving Strategies
- 15:05Monitoring and Observability
- 16:15Organizational Excellence and Team Collaboration
- 19:49Scaling Challenges and Optimization Strategies
- 22:16Security, Compliance, and Governance
- 24:10Future Trends in ML Inference Platforms
- 26:02Conclusion and Final Thoughts
related talks
29mMachine learning in production
04mInterview
31mUse-Case Driven ML Platforms
26mManaging ML Models @ Scale
24mBuilding an ML Application Platform from the Ground Up
32mTeam Teardown: Evolving an ML Platform
45mBuilding a Feature Platform to Scale Machine Learning
12mKeynote: Building Enterprise AI/ML Platform on Kubernetes