Hydro: A Stack for Faster Streaming Applications
summary
Hydroflow is a low-level streaming IR that compiles to Rust and runs each dataflow as a single-threaded program while composing across machines for distribution. The optimizer uses e-graphs (egg) to explore rewrite rules and finds nonobvious layouts; applying those transforms scales classic protocols (Paxos variants) up to about 5x throughput in experiments. Katara synthesizes verified CRDTs from sequential code using SMT-backed verification (and LLMs to speed synthesis). The stack is open source and the compiler runs in-browser at hydro.cc.
AI-written summary · about
chapters (13)
- 0:00Introduction to Hydro
- 1:04Problem with scaling cloud apps
- 2:28What is Hydro and its stack
- 3:52Hydroflow overview and design
- 7:20Stage programming in Hydroflow
- 9:25Hydroflow execution model
- 14:00EG graphs for optimization
- 20:41Optimizing folds in streaming
- 21:43CRDTs in distributed systems
- 24:29Katara for CRDT synthesis
- 27:58CRDTs for strong consistency
- 29:00Research and open-source resources
- 30:07Q&A session
