The TileDB Array Data Storage Manager
summary
TileDB claims an open source C++ storage manager for multi-dimensional sparse and dense arrays that maps n‑D cells into one‑dimensional tiled files, stores immutable timestamped fragments, and consolidates them for fast reads. It demonstrates S3-backed arrays (Azure/Google planned), parallel filters and compression, L1-cache–sized tiles, Intel TBB multithreading, multiple readers multiple writers, a Presto/Spark connector and a zero-copy NumPy integration used on LIDAR and financial datasets.
AI-written summary · about
chapters (17)
- 0:00TileDB introduction and vision
- 1:50Problems with storage formats
- 3:58Multi-dimensional arrays model
- 7:54TileDB vs DBMS
- 10:23TileDB storage features
- 13:29TileDB updates and snapshots
- 15:40Performance with filters and cache
- 17:26Cloud and local storage
- 18:31Thread safety and parallelism
- 20:18TileDB vs HDF5 Parquet
- 22:43Arrow integration and future
- 23:04Closing remarks
- 24:07Demo: LiDAR data with TileDB
- 27:24Demo: S3 SQL with Presto
- 30:15Q&A: Type system Python API
- 33:05Q&A: JavaScript reimplementation
- 34:58Q&A: Parallelism S3 performance
related talks
01h 10mThe TileDB Universal Database
30mHow to grow your own distributed array database
29mDistributed SQL How TiDB Modernized Data Management
29mDistributed SQL: How TiDB Modernized Data Management
47m10 Reasons Why the TiDB Database Should Power Your Next Development Project
45mThe Database: A Sophisticated and Special Data Structure
36mArcticDB, the OLAP antidote
29mMapDB - taking Java collections to the next level