Tower of Babel: Making Apache Spark, Kubeflow, and Kubernetes Play Nice
summary
Uses Apache Spark + Mahout to run a distributed SVD that lets denoising avoid a single-node 500 GB RAM requirement, reads DICOM CT lungs from S3 into PVCs and RDDs, and publishes a Kubeflow pipeline that reproducibly runs the stages. Engineering workarounds include copying shuffle files and spilling to object storage to cope with Kubernetes lacking a shuffle service and scale-to-zero limits. Pipeline successfully improves noisy low-dose CTs but rapid antigen tests and changing needs make the approach less useful in practice.
AI-written summary · about
chapters (8)
- 0:00Introduction and context
- 4:33Setting the scene
- 6:24Defining the problem and stack
- 12:52The pandemic case study
- 16:51Building the pipeline
- 23:23Results and lessons
- 27:14Optimization and alternatives
- 34:06Q&A and concluding remarks
related talks
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
26mBuilding Apache Spark Clusters on K8s for AI Workloads
29mScaling Apache Spark on Kube to Apple Scale
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
35mModern Big Data Pipelines over Kubernetes [I]
21mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
33mCompliant Data Management and Machine Learning on Kubernetes