The architecture evolution of Apache DolphinScheduler
summary
Apache DolphinScheduler traces architecture changes from 1.2 to 1.3 and a planned 2.0 that aim to cut DB pressure and latency by moving work off the database, adding a dispatcher, and replacing Zookeeper locks with a slot algorithmic assignment. Production footprints include handles 100k+ jobs/day across ~40 clusters and ~250 contributors. 2.0 refactors the master thread model, adds a SPI plugin system for custom tasks/connectors, implements fair task priority and reports measurable reductions in contention and faster scheduling in deployed environments.
AI-written summary · about
chapters (9)
- 0:00Introduction to DolphinScheduler
- 3:24Key features and advantages
- 9:47Real-world use cases
- 15:54Community and ecosystem
- 21:37Architecture evolution 1.x
- 26:11UI redesign and 2.0 goals
- 31:05Plugin system and extensibility
- 34:55Roadmap and future plans
- 39:11How to contribute
related talks
39mIntro: Scheduling SIG
34mKilling the Dinosaur: A migration story
45mEvolution of Spinnaker Internals & Operations
39mBuilding Scalable Orchestration
24mArchitecting the Unconventional: A Case Study in Software Design
38mFrom Flink 1.3 to 1.4
40mDeconstructing Apache Hadoop: A Dialogue About a Cloud-Native Refactor
47mClean Architecture Orchestration over Apache Camel