Suki: Choreographed Distributed Dataflow in Rust
summary
Suki is an embedded Rust DSL that uses staged programming to emit per-location Rust sources which compile to zero‑overhead binaries, relying on the hydroflow IR and a quoting macro Q! quoted expressions to mark cloud-side logic. The optimizer rewrites location boundaries (fusion, partitioning, pipelining) and achieves up to 2.5× throughput on Paxos/2PC workloads. Process specs generate Terraform for GCP VMs, the repo includes runnable examples, and the runtime exposes a send operator for typed networked streams.
AI-written summary · about
chapters (10)
- 0:00Suki introduction and goals
- 2:07Suki’s three design principles
- 3:53Suki’s location model and operators
- 4:55Stage programming and code generation
- 7:02Writing a Suki program
- 10:15Suki’s Rust streaming API
- 10:36Optimizing distributed dataflow
- 13:03Choreographies to cloud resources
- 14:47Conclusion and open-source
- 15:49Q&A
related talks
01h 02mHydroflow: A Compiler Target for Fast, Correct Distributed Programs
29mLessons Learned: Porting a Streaming Data Pipeline from Scala to Rust.
16mEmpowering Fearless Data Engineering
55mScaling out operational research with Rust
42mComposable Parallel Processing in Apache Spark and Weld
29mGouki Scheme: An Embedded Scheme Implementation for Async Rust
40mRust’s approach of getting things right
28mRust and Scala, Sitting in a Tree….