dbverse: Composable database libraries for larger-than-memory scientific analytics
summary
DB verse provides composable database libraries built on DuckDB and R that transpile familiar R generics into SQL (via the deer package and DVplier) and run lazily in-database. Benchmarks show DB Matrix handling a 30,000×1,000,000 sparse matrix and outperforming HDF5Matrix by up to an order of magnitude while avoiding memory failures; DB spatial uses DuckDB’s spatial extension to beat sf by up to 25× on polygon intersections; DB sequence queries a 28 GB BAM with similar or better performance than samtools and enables queries impossible in memory.
AI-written summary · about
chapters (6)
- 0:00Motivation and challenges
- 1:18Introduction to dbverse
- 2:00Syntax and implementation
- 3:00Performance benchmarks
- 4:30Conclusion and roadmap
- 5:19Q&A session
related talks
45mAnalytics for not so big data with DuckDB
25mA quack of all trades: DuckDB, a versatile analytical database to keep in your toolkit
28mAnalyze data easily with duckdb - and the implications on data architectures
24mData infrastructure to build bigger with less
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
18mduckplyr: Analyze large data with full dplyr compatibility
58mData Wrangling [for Python or R] Like a Boss With DuckDB
51mMotherduck: Data Analytics in the Post Big-Data Era