Orchestrating Scalable Data Pipelines with Apache Toree, YuniKorn, Spark, and Airflow
summary
A self service, multi tenant data platform combines Apache Toree with a Jupyter Gateway so notebook kernels spawn on a Spark cluster rather than local pods, scales by adding nodes, and uses Kubernetes plus the YuniKorn fair scheduler with separate interactive queues to avoid noisy neighbors so kernels become ready in seconds. Production pipelines export notebooks into Airflow via Elyra visual DAG editor, target Iceberg on S3 for storage, and rely entirely on open source components.
AI-written summary · about
chapters (7)
- 0:00Introduction to data pipelines
- 5:09Why notebooks for data engineers
- 10:16Apache Toree and kernels
- 13:30Scaling with Apache Spark
- 16:21YuniKorn for multitenancy
- 19:54Scheduling with Apache Airflow
- 22:25Platform components summary
related talks
26mRe-architecting the Data Platform with Spark on Kubernetes
42mBuilding Robust Streaming Data Pipelines with Apache Spark
17mDemocratising data analytics
33mSpark Operator - Feature Engineering with Spark on Kubeflow
35mBringing Your Data Pipeline into The Machine Learning Era
22mOrchestrate your data pipelines using Apache Airflow
34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh
20mBuild an e2e analytics application using DataHub