h5pydantic, from a Synchrotron through Python to HDF5
summary
h5pydantic maps Pydantic models to HDF5 so scientists declare group/dataset/attribute schemas in Python and then dump and load .hdf5 files without writing low‑level h5py code; it validates types, enforces required fields, encodes units, and embeds model versioning in files. It generates human‑readable layout docs (Sphinx → PDF) and exposes HDF5 features like per‑dataset compression. It is already used for synchrotron experiments to keep thousands of noisy measurements reproducible and to avoid late‑save failures via validators.
AI-written summary · about
chapters (10)
- 0:00Start
- 2:17Lots of metadata
- 6:54What Users and Scientists want
- 8:11HDFS Core Concepts
- 10:05Hand written, hand wavy hdf5 contents
- 11:48Hand written calibration
- 14:10Defining the model with h5pydantic
- 16:31Using the Model
- 20:17Pydantic Fields example
- 24:22Design
related talks
36mStore and manage data effortlessly with HDF5
25mpypet: A Python Toolkit for Simulations and Numerical Experiments
01h 20mAdvanced Data Storage
26mIntro to Pydantic, run-time type checking for your dataclasses
21mHiPy: Extracting High-Level Semantics From Python Code For Data Processing
02h 47mPython Workflows to Extract and Plot Satellite...
21mLet JSON Schema and Pydantic write your data models
01h 01mParallel Python Analyzing Large Data Sets