Harnessing WebAssembly for Real-time Stateless Streaming Pipelines

on YouTube ↗26m

" summary

Claims that WebAssembly lets Redpanda run WASM in-broker transforms safely and efficiently using WASI sandboxing, preallocated per-thread memory, and a 3,000 ms gas limit to prevent CPU blocking. Implements an SDK and CLI (rpk transform build) to compile Go/Rust/etc. into wasm, deploy once and replicate to all brokers; demo runs a distance-calculation feature-engineering transform, loads on three partitions and executes real-time, showing low-latency stateless pipelines inside the broker that actually work.

AI-written summary · about

" chapters (11)

1. 0:00Introduction and background
2. 3:00Data pipeline inefficiencies
3. 6:33WebAssembly overview
4. 9:43Safety and efficiency in brokers
5. 13:36User experience and SDK
6. 15:44Stateless pipelines explained
7. 17:10Demo setup and data context
8. 21:28Feature engineering in broker
9. 22:10Building and deploying WASM
10. 23:40Live transformation demo
11. 25:28Q&A

" related talks

• 25mExtending a High-performance Data Streaming System with WebAssemblyTyler Rockwood · Cloud Native Wasm Day NA 2023
• 50mWeb AssemblyBrian Sletten · Developer Summit 2020
• 31mRunning AI workloads with WASM - is it production ready? [Panel]Shivay Lamba, Michael Yuan, Saiyam Pathak, Saloni Narang · Wasm I/O 2025
• 48mWebAssembly Beyond the BrowserMete Atamel · JCON Europe 2023
• 35mWebAssembly beyond the browserMete Atamel · code.talks 2023
• 33mWebAssembly: The Next Frontier of Cloud ComputingEdoardo Dusi · containerday 2023
• 26mWebAssembly for a Reactive Internet of ThingsColin Breck · Reactive Summit 2022
• 39mFrom Event-Driven to AutomotiveP J Laszkowicz · Wasm I/O 2023