Using embedded SQL engines for plotting massive datasets on a laptop
summary
Claims that a laptop can plot larger‑than‑memory data by pushing aggregations and percentiles into SQLite or DuckDB, computing boxplot percentiles and histogram bin counts with SQL, then handing results to Matplotlib. Demonstration runs on ~4.5M NYC taxi rows, shows memory usage 0.3 GB vs 1.2 GB when compared to pandas, implements percentile queries and binning, provides an open source package (pip install QP SQL) and working code so the method actually produces plots without swapping or clusters.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:28Traditional in-memory problems
- 4:02Why embedded SQL engines
- 6:10SQLite and DuckDB overview
- 9:36Box plot generation with SQL
- 15:35Histogram generation with DuckDB
- 18:46Implementation and demos
- 21:17Memory efficiency vs pandas
- 23:06Q&A
related talks
25mProcessing Biggish Data with DuckDB and Python
30mDuckDB: Bringing analytical SQL directly to your Python shell
45mAnalytics for not so big data with DuckDB
01h 02mAnalytics for not-so-big data with DuckDB
24mBuild your own data warehouse for personal analytics with SQLite and Datasette
37mAccelerating Python Analytics by In-Database Processing
03h 07mData of Unusual Size: Interactive Visualization
32mScalable Python and SQL Data Engineering without Migraines