Streamling: Lightweight, Extensible Streaming on DataFusion
summary
A Rust streaming engine built on Apache DataFusion and Arrow turns Kafka Avro into continuous Arrow RecordBatch streams using DataFusion’s planner and custom TableProviders, adds Chandy–Lamport marker→ack→finalizer checkpoints with at‑least‑once guarantees (state in memory, SQLite, or Postgres), and exposes WASM via Extism plus abi_stable FFI plugins and dynamic lookup tables for enrichment. It deliberately avoids distributed shuffle and stateful joins, boots pipelines in about 5 second startup versus Flink’s minutes, provides live inspect UI and Kubernetes control/data‑plane deployment, and is now open source.
AI-written summary · about
chapters (11)
- 0:00Introduction to streaming
- 4:24Moving from Flink to Streamly
- 7:57Why build Streamly
- 12:14Streamly engine internals
- 16:30Extending beyond SQL
- 21:24Observability in Streamly
- 23:32Building the product Turbo
- 27:50Kubernetes deployment
- 31:48Audit and state management
- 35:28Open sourcing Streamly
- 36:32Q&A
related talks
29mLessons Learned: Porting a Streaming Data Pipeline from Scala to Rust.
34mStreaming video analysis in Rust using Pravega
39mChallenges of Building Blazing Fast Data APIs
16mEmpowering Fearless Data Engineering
25mVector Streaming: The Memory Efficient Indexing for Vector Databases
22mAntimony: a real time distributed stream processing system
28mRust and Scala, Sitting in a Tree….
23mSparrow, Pirates of the Apache Arrow