Enterprise-grade Full Stack ML Platform
summary
Describes an enterprise ML platform built at Dell using Metaflow and Outerbounds that claims to let data scientists run end-to-end workflows without deep SRE work by handling data, compute, orchestration, and versioning. Demonstrates a 25 gbps S3 connector, automatic snapshotting and automatic versioning, and the ability to spawn ~100 parallel containers on Kubernetes for large hyperparameter sweeps. Integrates with Argo, Airflow and AWS Batch and reports reproducible runs and deployed pipelines in production.
AI-written summary · about
chapters (17)
- 0:00Enterprise ML infrastructure challenges
- 2:11ML production needs robust foundation
- 3:35Data exploration for data scientists
- 5:20Orchestration and collaboration in ML
- 8:15Platform engineering and security
- 10:19Build vs buy for ML platforms
- 11:23Metaflow introduction and Netflix journey
- 13:06ML vs traditional software
- 14:50ML infrastructure priorities
- 16:18Metaflow DAG workflow design
- 17:22High-performance data access
- 18:24Automatic versioning and reproducibility
- 19:50Scaling training with parallel compute
- 21:12Deploying Metaflow to orchestrators
- 23:17Security and privacy in Metaflow
- 25:04Glue layer for ML infrastructure
- 27:51Q&A
related talks
36mHuman-Friendly, Production-Ready Data Science Stack with Metaflow & Kubernetes
09mPlatform Engineering for Data Science at Scale
26mManaging ML Models @ Scale
32mHuman-Friendly, Production-Ready Data Science Stack With Metaflow & Kubernetes
30mML Platform for Insurance Conglomerate
33mDiscover How to Create Your Own Metadata-Driven ML Platform from Scratch
37mPlatform for Complete Machine Learning Lifecycle
31mMLflow: Platform for Complete Machine Learning Lifecycle