Embrace the chaos: Composable data systems with fewer asterisks
summary
Concrete claims and demos lead the choice: adoption of Apache Arrow as a zero copy in‑memory column format, using ADBC + FlightSQL for driver and RPC connectivity, shrinks glue code and avoids transpose costs. A DuckDB benchmark drops a million‑row roundtrip from ~28s to ~7s, and Streamlit removal of 1,000 lines of custom code shows real maintenance wins. The stack actually runs: Arrow memory layout, ADBC drivers, and Flight RPC interoperate and deliver measurable speed and composability.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:26Building vs off-the-shelf
- 4:14Composable data systems layers
- 7:25Standards for interoperability
- 11:22Five criteria for standards
- 15:40Modern data stack standards
- 18:31Apache Arrow format
- 21:20Arrow adoption and ecosystem
- 23:07Query plans and Arrow compute
- 23:51ADBC for connectivity
- 25:35Data hierarchy needs
- 27:01Avoiding one-off solutions
- 28:47Reducing edge cases
- 30:12ADBC and Flight SQL
- 33:47Customization and swapping
- 34:10Simplifying complex problems
- 35:37Q&A
related talks
58mComposable Data Systems, the new frontier
33mArrow in Flight - New Developments in Data Connectivity
35mPractical Applications of Apache Arrow
01h 04mWhere We’re Going, We Don’t Need Rows: Columnar Data Connectivity with Apache Arrow ADBC
42mState of the Apache Arrow Ecosystem:How your project can leverage Arrow!
34mLeveraging Apache Arrow for ML Workflows
40mDuckDB: Harnessing in-process analytics for data science and beyond
32mThe expanding Apache Arrow universe