Data persistence with consistency and performance in a truly serverless system
summary
ArcticDB is a pure Python library that uses commodity object storage (S3/minio) and a few concrete techniques to get ACID‑like behavior without servers: structured keys and immutable blocks avoid in‑place corruption, event‑streaming version keys provide time travel, and a tiny mutable head key speeds lookups. Persistent B‑trees, background compaction, and an ECS C++ vectorized engine power a QueryBuilder that filters 10 million rows in ~66 ms, multiplies a column in ~81 ms and joins in ~94 ms on a 100,000‑string cardinality test.
AI-written summary · about
chapters (15)
- 0:00Arctic DB introduction
- 1:10Data storage evolution
- 4:20OLAP server challenges
- 6:23Hybrid store limits
- 8:49Immutable storage streaming
- 10:13Arctic DB design Python
- 11:37Structured keys data structures
- 14:02Version tracking lists
- 15:48Optimizing object store
- 17:34Arctic DB features
- 19:19In-memory query builder
- 20:43Performance benchmarks with 10 million rows
- 22:50Advanced joins transforms
- 24:59Future roadmap
- 26:42Closing resources
related talks
34mData Wrangling with DuckDB
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
39mPython and SQL: Better Together, Powered by DuckDB
25mProcessing Biggish Data with DuckDB and Python
34mBridging Pandas and Polars
40mPyData Stack: Pure Python open source data platforms
28mBridging Python and Apache Iceberg: The Power of PyIceberg
30mPython and the AI Lakehouse