Faster SQL with pandas and Apache Arrow
summary
Pandas connects to databases using new Apache ADBC drivers that pass Arrow columnar data and move work into compiled code so SQL I/O gets both correct types and far higher throughput. Read_sql with dtype_backend="pyarrow" preserves nullable integers and reproduces database 32‑bit integers for true roundtrips. Benchmarks show default ODBC inserts around 22s versus ADBC at 658 ms (roughly 35× faster) without custom copy hacks. Drivers for Postgres SQLite BigQuery and Flight SQL appear on PyPI and require pandas 2.2 / ADBC 0.8.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 2:57Pandas SQL drivers limits
- 7:58Data transfer mismatch
- 10:47Pandas data type issues
- 16:29to_sql performance issues
- 21:26Arrow Database Connectivity
- 23:55ADC missing value handling
- 25:22ADC performance gains
- 27:07Arrow rich data types
- 29:41ADBC drivers overview
- 33:32ADBC development guide
- 34:36Flight SQL integration
- 35:40Q&A
related talks
31mODBC takes an Arrow to the knee: ADBC
01h 04mWhere We’re Going, We Don’t Need Rows: Columnar Data Connectivity with Apache Arrow ADBC
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API
33mArrow in Flight - New Developments in Data Connectivity
32mThe PyArrow revolution in Pandas
29mFaster PySpark with Apache Arrow
24mPolyglot data with python: Introducing Pandas and Apache Arrow
33mturbodbc: Turbocharged Database Access for Data Scientists