Modern Data Science in a Cloud Native World
summary
Demonstrates using Pachyderm data versioning on Kubernetes to build declarative pipelines that expose files to containers via PFS, shard inputs and run parallelized containers, spin pods up/down, and automatically reprocess only changed commits. Implements an Iris example that trains and evaluates 77 models trained in parallel, pushes built images to a registry, and ties pipeline updates into CI/CD so new model commits trigger incremental pipeline execution that completes end-to-end.
AI-written summary · about
chapters (11)
- 0:00Start
- 0:22Code and Slides
- 1:23The Data Science Pipeline
- 2:52Pipelines
- 5:07Containerization
- 6:26Distributing Workloads
- 7:03Data Version
- 7:22Reproducibility
- 8:33Data Provenance
- 10:27Summar
- 11:26Demo
related talks
33mCompliant Data Management and Machine Learning on Kubernetes
34mPachyderm: Unlock the Power of Kubernetes for Big Data
39mEnd to End Machine learning pipelines for Python driven organizations
19mScalable Data and AI Solutions with Cloud-Native Architecture
28mDevOps for Data Science With Kubernetes
21mMastering the DevOps Machine Learning Pipeline
46mDeploying scalable Machine Learning models with Kubernetes
30mContainer workflows for data science and ML