Compliant Data Management and Machine Learning on Kubernetes
summary
Claims a reproducible, auditable ML pipeline on Kubernetes that records full data provenance and works at scale; uses Pachyderm for git-like dataset versioning, Kubeflow for orchestration and TF training, Seldon for serving, plus an object store (Rook) and KVC for caching. Demo trains an issue-summarization model, builds a job-tagged Docker image, and traces an inference back to the exact commits and preprocessing steps; notes real clusters hitting 1,500+ nodes and hundreds of terabytes.
AI-written summary · about
chapters (23)
- 0:00Intro
- 0:10Overview
- 1:16Agenda
- 1:48What is compliance
- 3:04Google autocomplete
- 4:50Compliance
- 6:05Guidelines
- 8:38Framework
- 9:30Versioning
- 11:44The good news
- 12:19Project overview
- 13:37CubeFlow
- 14:26Pachyderm
- 16:18Pachyderm Integrations
- 16:57KVC Standards
- 19:26Existing Projects
- 20:21Demo
- 21:07Pipeline
- 21:31REST API
- 22:35Pods
- 23:50Pipelines
- 27:39Acknowledgement
- 28:53Questions
related talks
30mRealizing End to End Reproducible Machine Learning on Kubernetes
39mEnd to End Machine learning pipelines for Python driven organizations
37mA Day in the Life of a Data Scientist. Conquer ML Lifecycle on Kubernetes
46mDeploying scalable Machine Learning models with Kubernetes
28mMachine Learning on Kubernetes at Shell: A Kubeflow Journey
35mOrganizing data engineering around Kubeflow
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
19mModern Data Science in a Cloud Native World