Human Friendly, Production Ready Data Science Stack
summary
An open‑source stack from Netflix called Metaflow provides concrete, production features: Python @step decorator to define DAG nodes, automatic snapshotting and versioning of data and artifacts, and a fast data layer that can stream 25 Gbps from S3. Users run local code with pip install metaflow and transparently scale to cloud compute (AWS Batch, Kubernetes) with per‑step resources (examples: 100 parallel train tasks on 128 GB instances, scaling to thousands of tasks). Integrates with existing orchestrators and reproduces failures locally.
AI-written summary · about
chapters (16)
- 0:00Data science infrastructure challenges
- 1:08Why ML projects fail in production
- 3:15ML productivity missing link
- 4:58Data scientist workflow
- 8:08ML workflow orchestration challenges
- 13:46Metaflow for human-friendly ML
- 15:12Metaflow decorators for workflows
- 15:33Efficient ML data access patterns
- 16:37Automatic versioning and state management
- 17:39Scaling compute with resources
- 18:42Multi-cloud and Kubernetes integrations
- 19:46Scheduling workflows with orchestrators
- 21:10Enterprise collaboration and isolation features
- 22:36Provisioning execution environments
- 22:58Infrastructure layer cake for ML
- 23:41Closing and community
related talks
32mHuman-Friendly, Production-Ready Data Science Stack With Metaflow & Kubernetes
44mHow Metaflow Became Netflix's Beloved Data Science Framework
46mThe Python Data Ecosystem: Navigating a fragmented landscape
28mBuilding Self-Service Data Platforms
43mData Pipelines à la Mode
29mStreamlining Competitive Data Science at CERN: Running ML...
29mHow we turn data teams into better engineers at Netflix
33mHow to build production-ready data science pipelines with