Onyx: Distributed Workflows for Dynamic Systems
summary
Onyx is a Clojure distributed computation system that reifies workflows as maps and plain functions, uses a catalog + plugins to swap inputs like core.async or kafka, and exposes a state API for lifecycle hooks. It wires HornetQ for messaging and ZooKeeper for coordination, claims hornetq can hit millions of messages per second per node, supports hybrid batch/streaming transactions, includes an atomic (Datomic) plugin, and the demo runs and uppercases records end-to-end.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:24What is Onyx?
- 1:09Onyx Goals
- 3:35The Problem
- 6:08Cascading: A Solution?
- 11:26Storm: A Solution?
- 11:52Cascalog: A Solution?
- 12:16Decomposing the Monolith
- 12:33Guiding Example
- 12:50Data Representation
- 14:28Functions
- 15:01Did Somebody Say Transducers?!
- 15:26Positional Representation
- 16:36Execution Context
- 18:29Extensibility Model
- 20:30Runtime Discovery
- 23:58Task State API
- 26:56Architecture Overview
- 28:03Wait! That's Slow!
- 32:49Questions?
related talks
39mWorkflows, a new abstraction for distributed systems
19mDistributed Workflows for Microservices-Style Applications [I]
30mRunning multiple Nubanks at once
22mManaging Distributed Workflows at Scale
34mDeveloping Asynchronous Polyglot Applications with vert.x
01h 01mHow to build real-world applications with Orleans
54mReacting to an Event-Driven World
43mCommander: Better Distributed Applications through CQRS and Event Sourcing