Timely Dataflow
summary
Timely Dataflow presents an implementation of Naiad’s low-latency cyclic dataflow model in Rust with an unsafe serialization layer for performance; examples show exchange-based shuffles, probe for coordinated progress, and differential dataflow for incremental updates. Public benchmarks compare single-thread and 128-core systems and claim orders-of-magnitude gains; incremental graph queries update in microseconds for small changes. Code is embedded Rust, no ready-made connectors, and the system trades hand-crafted operators for high throughput and low latency.
AI-written summary · about
chapters (12)
- 0:00Intro
- 1:54Why bother
- 2:50What is it
- 5:31Performance
- 7:30Canvas
- 9:50Partitioning
- 11:42Flexibility
- 12:45Differential Flow
- 14:07Mobius Inversion
- 15:19Differential Dataflow
- 16:12Nyad
- 17:06Questions
related talks
01h 01mLogic and Algebras for Distributed Computing: Applications and Open Questions
32mReactive Datalog for Datomic
17mBring Your Own Data Structures to Datalog
05mLightning Talk: A Fast, Concurrent Data Loader for Time-Series Data
52mData Over Code: A Short Hyperbole
22mGraphite-Rust: A Fast Time Series Database Implementation
45mBeyond Lists: High Performance Data Structures
21mDesigning Modular Data Pipeline