Scaling Kubeflow for Multi-tenancy at Spotify

on YouTube ↗32m

" summary

Spotify runs Kubeflow Pipelines on GKE with a Python SDK and TFX, serving 60+ teams and ~600 users, logging 30,000 models and 100,000 pipeline hours (~300 runs/day). Platform uses Terraform + kustomize for cluster and namespace provisioning, a custom controller that maps team profiles via Google groups to Kubeflow namespaces, and a multi-cluster versioning strategy (per-SDK major-version clusters) to avoid breaking changes. Deployments use Argo CD and CI, observability adds SLOs in GCP and a custom kubeflow-state-metrics exporter for product-specific Prometheus metrics.

AI-written summary · about

" chapters (28)

1. 0:00Intro
2. 1:48Kubeflow Platform for ML
3. 2:28ML Workflow @ Spotify
4. 4:48Kubeflow clusters
5. 5:30Kubeflow platform stats
6. 6:15Our solutions
7. 6:58Team based multi-tenancy
8. 9:03Team profile management
9. 10:37Team profile example
10. 12:49Existing setup & problems
11. 14:12Multi-cluster strategy illustration
12. 15:09Benefits of multiple clusters
13. 15:57Multi-cluster implementation
14. 16:53Multi-cluster based Kubeflow Platforr
15. 17:48New challenges
16. 18:53Solution: ArgoCD
17. 19:36Multi-cluster CD
18. 20:39Multi-cluster reliability
19. 21:34"Infrastructure"-as-code
20. 23:36SLO Tracking
21. 24:45Telemetry and Metrics
22. 25:34kubeflow-state-metrics
23. 26:46Infrastructure and Infrastructui
24. 28:18Product Identity
25. 29:22Expanded Observability
26. 29:56Expanded "On-Cluster" Compute
27. 30:40Self-Service
28. 31:14Open Source!

" related talks

• 38mBuilding and Managing a Centralized Kubeflow Platform at SpotifyKeshi Dai, Ryan Clough · KubeCon + CloudNativeCon North America 2019
• 34mKubeflow: Multi-Tenant, Self-Serve, Accelerated Platform for PractitionersKam Kasravi, Kunming Qu · KubeCon + CloudNativeCon North America 2019
• 30mTeam Teardown: Driving Platform Adoption and SuccessMaya Hristakeva, James Kirk, Lex Beattie, Maisha Lopa, Samuel Ngahane · TWIMLcon: AI Platforms 2021
• 54mScaling Machine Learning Pipelines in CloudSalman Iqbal · NDC Oslo 2021
• 21mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)Holden Karau, Trevor Grant · Open Source Summit + Embedded Linux Conference North America 2019
• 51mMachine Learning using Kubeflow and KubernetesArun Gupta · Devoxx Belgium 2019
• 31mKubeflow - Data Science on SteroidsSascha Grunert · ContainerDay 2019
• 36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)Trevor Grant · KubeCon + CloudNativeCon + Open Source Summit China 2019