Array Time travel with version HDf5
summary
Versioned HDF5 implements versioned HDF5 arrays by wrapping h5pi and exposing per-version virtual datasets that map into immutable raw chunk stores. Each chunk is content‑addressed with SHA‑256 chunk map so unchanged chunks are reused like Git; inserts/deletes force chunk shifts, so heavy reordering can be ~10–20x slower, while appends and updates are cheap. It exposes bi‑temporal tables (valid time + transaction time), lets computations pin a transaction timestamp for consistent reads, and is open source on GitHub.
AI-written summary · about
chapters (4)
- 0:00Start
- 9:25THE PROBLEMS WITH DATA FLOW GRAPH COMPUTAIONS If data changes over time, how can we: - Achieve consistent results if clata changes during computation? Explain results if stored data has already changed? Store data in a temporal database
- 10:25TEMPORAL DATABASES A temporal database records how data changes through time Valid time: the time a fact became true in the real world Transaction time: the time a fact was recorded in the database. Unitemporal (one time axis) or bitemporal valid and transaction time
- 19:22USING CHUNKED STORAGE TO IMPLEMENT VERSIONING 1. Copy-on-write: chunks are immutable 2. Write new copy of any modified chunks 3. Create a virtual dataset for each version: a view of raw chunks
related talks
19mMultiverse Notebook: Shifting Data Scientists to Time Travelers
33mPortable, type-fancy multidimensional arrays
50mFast Analytics On Big Data
38mUncluster Your Data Science Using Vaex
28mBreaking up arrays up into chunks for fun and science with Xarray and Dask
53mCan Haskell Become a Great Language for Data Science?
45mMassive distributed processing with H2O
30mXarray DataStructures in Biology: Examples and Best Practices