A Scientist's Guide to the Relational Data Model
summary
Practical claim: expose custom scientific files as SQLite virtual table so tools like Python R Paraview and ggplot can query without copying. Implements a C virtual-table reader for a Fortran heat sim, adds an indexed time queries path and shows no ingest streaming into data frames. Reports Paraview with ~150 readers, custom formats producing terabytes to petabytes, and successful runs on hundreds of gigabytes tested with indices but warns SQLite is interpreter‑based and can hit scale limits. Repository and notebooks are provided.
AI-written summary · about
chapters (16)
- 0:00Start
- 2:50Large Scale Simulations
- 3:21Custom Data Formats
- 3:57Open-Source Scientific Visualization Tools
- 4:53Typical Analysis Workflow
- 9:20Relational Data Model
- 10:55Create Virtual Table
- 11:27Plotting
- 16:10Deep Liar
- 21:26Indices
- 22:00Iterator
- 23:12Sequel Lite Code
- 23:33Best Index and Filter
- 50:22Heat Equation in Spark
- 51:21Array Indexing
- 52:33Bonus Round
related talks
02h 46mData analysis with SQLite and Python
24mBuild your own data warehouse for personal analytics with SQLite and Datasette
51mData tiering using data Virtualization in SQL
20mGive me the Data
19mDatabases using R The latest
23mFANTASTIC DATABASES AND WHERE TO FIND THEM
27mUsing embedded SQL engines for plotting massive datasets on a laptop
15mDatasette and Datasette Publish