A Clojure DSL for defining CI/CD orchestrations at scale
summary
Carson is an event driven orchestration engine built at Oracle that uses a Clojure DSL to define pipelines as directed acyclic graphs, dispatching jobs across Jenkins/Hudson, Artifactory, Mesos, Docker and messaging like Kafka or RabbitMQ. It handles about 100,000 events/day, runs roughly 18,100 orchestration instances/day, and stores 500 orchestration definitions. The DSL provides macros, multimethod predicates, def-start/def-transition handlers, a test harness, D3 visualizations and async job dispatchers that actually submit Mesos tasks and promote Artifactory artifacts.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 1:00Agenda
- 1:53History
- 3:51Scale
- 5:40Stakeholders
- 8:31What is it
- 10:28Events
- 11:42Architecture
- 16:11The scale
- 17:33What is an orchestration
- 18:54Basic event handler definition
- 20:43Basic handler types
- 21:46DefStart
- 25:20DefTransition
- 27:15Graphs
- 28:33Another DSL
- 30:13Mesos Task
- 34:28Why Clojure
- 38:08Action Helpers
- 39:44Job dispatching
- 41:05Vault
- 42:41Oxygen
- 43:01Future plans
- 44:06Summary
related talks
47mClean Architecture Orchestration over Apache Camel
35mJenkins Scaling and Organization for an Efficient CI
46mConcourse: CI/CD, 2020 edition
29mCI That Scales With your Project
19mFrom CDEvents To Actions: Designing the Workflow Conductor
34mContinuous Deployment at Enterprise Scale
51mScala At Netflix
29mConcourse in the Real World: A Case Study in CI/CD and DevOps - Greg Meyer & Bryan Kelly, Cerner