Composable Data Systems, the new frontier
summary
Argues that replacing monoliths with interoperable pieces yields big wins: storage, planner and engines communicate via Apache Arrow and Substrait, connectivity uses Flight/Flight SQL and ADBC, and engines return columnar Arrow results with zero-copy transfers. Shows real speedups (benchmarks where Flight+Arrow far outperforms pandas+PyMySQL), cites Apache Druid adopting Flight, and demos a working 50-line SQL engine built with Ibis, Substrait, Arrow and Asero.
AI-written summary · about
chapters (20)
- 0:00Introduction and keynote setup
- 2:09Evolution of data systems
- 9:44Problems with monolithic systems
- 16:26Composable data systems defined
- 21:00Query interfaces and abstraction
- 23:07Substrait intermediate representation
- 29:19Connectivity protocols and Flight
- 34:01Apache Arrow format benefits
- 39:47Arrow-based compute engines
- 42:19Data storage and formats
- 44:27Scalability and performance gains
- 45:32Building a SQL database in 50 lines
- 47:22Q&A session begins
- 47:43Tradeoffs of composable systems
- 48:49Open source standards and vendor lock-in
- 50:35Why Arrow is fundamentally faster
- 52:22Zero-copy conversion vs native APIs
- 54:34Cloud provider support for composable systems
- 56:19Handling complex types in Substrait
- 57:42Conference wrap-up and closing remarks
related talks
36mEmbrace the chaos: Composable data systems with fewer asterisks
33mArrow in Flight - New Developments in Data Connectivity
01h 02mApache Arrow: High-Performance Columnar Data Framework
01h 04mWhere We’re Going, We Don’t Need Rows: Columnar Data Connectivity with Apache Arrow ADBC
42mState of the Apache Arrow Ecosystem:How your project can leverage Arrow!
35mPractical Applications of Apache Arrow
30mRetooling for a Smaller Data Era
28mBridging Python and Lakehouse worlds with Arrow Flight and DuckDB.