Building WASM Powered Distributed Stream Platform
Talkat Cloud Native Wasm Day Europe 2022 · on Monday, 2022-05-16· at 15:15· Pavilion 3, Room F | Level 2 | Central Forum, Feria Valencia, Spain + Live Stream·
summary
Fluvio embeds SmartModule WASM filters and maps (Rust-based with proc-macros) into the data path so compute runs next to storage (append-only immutable streams) instead of moving raw payloads; batch blobs are copied into WASM memory via an alloc/unlock pattern, deserialized, and processed, enabling a reported 50x bandwidth reduction and >10x memory footprint versus Kafka. Platform supports ARM and x86 edges, exposes store bindings (filter, map, aggregate, join planned), and currently provides at-least-once delivery.
AI-written summary · about
chapters (23)
- 0:00Intro
- 0:15Agenda
- 0:56Overview of Fluvio
- 2:18Real-time economy
- 3:27Classical Data Processing
- 4:36Stream Processing Model
- 5:39Composability
- 6:33Moving data is not free
- 7:23Costly Stream processing
- 8:21Data movement is not secure
- 9:06Solution: Reduce data movement
- 11:17Edge Smart Stream processing
- 11:56Compute for Smart Streaming
- 12:50WASM: True Movable Compute
- 14:24Flattening into Unified Stack
- 16:06Fluvio: SmartModule
- 18:38Filter Binding
- 22:37Copying into WASM memory
- 25:31Map Binding
- 25:58Aggregate Binding
- 26:27Current store bindings
- 26:43Future Store bindings
- 28:02Improve Store Binding
related talks
25mExtending a High-performance Data Streaming System with WebAssembly
40mStream Processing and New Approaches: Edge Processing
39mAdvanced Stream Processing on the Edge
33mfStream: a managed stream processing platform at Flipkart
28mSweet Streams Are Made Of These
22mPlatform Engineers assemble! Create your FaaS Platform using WASM
49mscale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing Applications
28mDistributed Computation with WASM and WASI