Bridging the Chasm Between Research & Software Development
summary
A data-heavy fertility product with 200,000 users and huge nightly sensor streams faces repeated reimplementations of data handling and scattered “glue” code. The team extracts infrastructure glue into shared glue libraries, runs the Python processing modules on AWS instead of rewriting for Spark, and adopts MLflow model registry plus scikit-learn‑style pipelines (pickled models, reproducible envs) so trained models deploy reliably. Kubeflow is evaluated but MLflow wins for experimentation; retraining workflows and governance still require ML‑engineer support.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:40Agenda
- 1:53Context
- 7:27The problem
- 9:28The problem: Example 2
- 12:02Solutions
- 35:31Remaining challenges
- 38:04Outro
related talks
31mTransforming Data Science at PepsiCo: The Kubeflow Revolution
31mMLflow: Platform for Complete Machine Learning Lifecycle
35mOrganizing data engineering around Kubeflow
24mMLflow: An open platform to simplify...
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
18mAutomate the boring ML stuff with pipelines
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
14mKubeflow: Where ML Meets Kubernetes