GizmoEdge: A distributed DuckDB engine for IoT
summary
Distributed engine built on DuckDB that runs on anything from iPhone to Kubernetes, uses Apache Arrow IPC transfers and websockets to move columnar results, and parses SQL with a Postgres parser to rewrite aggregates (averages become sums+counts). Prototype is Python, shards are pre-built, and a live demo returns correct results from a pre-sharded trillion-row demo by dispatching work to ~100–1,000 workers, verifying shards with SHA256/MD5 handshakes, and collapsing partial results server-side; repo is not yet public and production hardening remains.
AI-written summary · about
chapters (13)
- 0:00Introduction to Gizmo Edge
- 1:16The challenge of edge data
- 2:13Architecture and design
- 3:59Communication and protocol
- 6:38Worker execution details
- 7:33Distributed aggregation
- 8:20Shard management
- 9:13Targeted broadcast sharding
- 10:49Tech stack and deployment
- 12:13Why DuckDB is ideal
- 13:12Live demo of Gizmo Edge
- 20:05Future roadmap
- 21:22Q&A and concluding remarks
related talks
28mAnalyze data easily with duckdb - and the implications on data architectures
40mDuckDB: Harnessing in-process analytics for data science and beyond
27mState of the Duck
34mData Wrangling with DuckDB
25mA quack of all trades: DuckDB, a versatile analytical database to keep in your toolkit
27mThe Duck and the DataFrame: A Data Engineer’s Journey with DuckDB
47mGábor Szárnyas | Data Track | Compass AI & Data Summit 2025
01h 02mAnalytics for not-so-big data with DuckDB