Dagobah, a Data centric Meta scheduler
summary
Dagobah is a data‑centric meta‑scheduler that treats each node as an addressable computation, records exact lineage and supports optional pinning to build/branch/commit. It runs PigPen on Hadoop, Spark on various platforms, and Docker containers on Titan, stores metadata in Datomic (backed by Cassandra), and memoizes outputs so multiple users share memoized immutable data instead of duplicating work. Deployed at Netflix it handles about 200 requests/day (~1,000 jobs/day), 13,000 node defs, ~200 unique addresses and 60,000 persisted data artifacts after six months.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:30Motivation
- 3:29Traditional Workflows
- 5:06Data Workflows
- 8:55Nodes
- 9:33Dagobah
- 14:51Request Definition
- 15:55Node Definition
- 18:59Deployment
- 21:50Resiliency
- 23:01Structural Sharing
- 26:16Bad Data
- 27:23Gates
- 28:21Actions
- 30:02How it works
related talks
47mDagster: An open source Python library for building data applications
29mDagster: a Framework for Data Processing Applications
49mData Platform Architecture, Evolution, and Philosophy
47mData - The Land DevOps Forgot
15mSPADE: Signal-Aware DAG Scheduling and Dynamic Provisioning for Data Processing Clusters
01h 01mData Infrastructure in a Multi-Cloud Environment
28mRevolutionizing the Big Data Age With Compute over Data
27mData Mesh for the Cognitive City