How to grow your own distributed array database
summary
A custom array store called Doc Brown holds tens of terabytes of raster data (one dataset ~5–6 TB) as chunked, per-dimension slabs so queries touch few chunks; chunks are deduplicated by hash and wired into an inverted index to provide immutable versioned snapshots, making small daily updates cheap. The library (not a service) uses pluggable backends — memory, filesystem, SQLite and DynamoDB in production — does optimistic concurrency, caching and compression, and the demo ingests 250 MB and reads via netcdf4 in Python.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:19What is data
- 4:16How to store data
- 9:17Our needs
- 10:59Demo
- 11:59Key Ideas
- 12:48Metadata
- 14:26Chunking
- 16:33Slices
- 17:30Versioning
- 17:58Indexing
- 21:59Library
- 26:49Readwrite session
related talks
37mThe TileDB Array Data Storage Manager
36mArcticDB, the OLAP antidote
43mWhy We Built Our Own Distributed Column Store
37mKEYNOTE: Data Architecture Turned Upside Down
25mManaging changes to your data with dat
25mOffline-first data: Getting Bigger
30mCatching Waves with Time-Series Data
30mWhat a serverless database means for users