Integrating Apache Arrow on WASM using Component Model
summary
Describes integrating Apache Arrow with WASM Component Model inside a Stateful Dataflow (SDF) streaming engine that runs each operator as a typed WASM component. Compares three concrete approaches: shared memory, copying buffers, and Arrow-as-resource, with tradeoffs (shared memory lacks broad language support, copying needs adapters, resource handles avoid per-language adapters). Claims columnar Arrow gives 10–100x speedups for analytics, implements a live demo that computes top‑5 vehicle speeds from Kafka streams and shows the pipeline running successfully.
AI-written summary · about
chapters (9)
- 0:00Apache Arrow and WASM intro
- 2:06Component Model for data stacks
- 4:15Stateful Data Flow engine
- 6:40Integrating Arrow in WASM challenges
- 9:47Sharing memory with Arrow
- 13:23Copying Arrow buffers
- 16:32Arrow as Component Model resource
- 22:47Live demo Arrow in WASM
- 27:05Future directions & Q&A
related talks
36mEmbrace the chaos: Composable data systems with fewer asterisks
25mExtending a High-performance Data Streaming System with WebAssembly
08mKeynote: WASI Standards and the Component Model: Transforming Application Development Forever
31mRunning AI workloads with WASM - is it production ready? [Panel]
28mDistributed Computation with WASM and WASI
26mHarnessing WebAssembly for Real-time Stateless Streaming Pipelines
31mBuilding WASM Powered Distributed Stream Platform
52mWebAssembly: from browsers to high load