Using Declarative Configs for Maintainable Reproducible Code
summary
Claims that declarative configs let large experiments stay maintainable and reproducible by separating parameters, verifying usage, and migrating old configs into the current schema. Concrete tools shown: yaml files for inputs, schema classes via attrs, and conversion with cattrs plus type checks with mypy/nbqa and CLI with typer. The team runs multi‑terabyte pipelines on HPC clusters with training runs taking ~two weeks, built WebKnossos and an automated pipeline, and demonstrates migrating old experiment configs so they still run on current code.
AI-written summary · about
chapters (15)
- 0:00Start
- 0:18Introduction & Problem Domain
- 3:02Goals: Maintainability & Reproducability
- 4:16Toy Experiment
- 5:07Declarative Configuration Exctraction
- 6:08Input Formats, Representations & Deserialization
- 8:49Landscape Overview
- 10:10Toy Example
- 11:08Split Configuration
- 13:46Type Checking
- 15:15Complex Example with Nested Configurations
- 18:45Evolution of Old Configurations
- 20:15Schema Versions & Evolutions
- 21:04Experiment Tracking
- 21:34Summary
related talks
18mEverything-as-Code: It's time to fix configurations
50mNode Project Configuration Management at Scale
30mFailing Fast with Data: Bringing CI to the Land of Data
41mConfigman - the grand unified theorem of configuration
32mConfiguration Is (riskier than?) Code
27mRun Your CI in Code, Not YAML
50mWhy You Need a Software Delivery Machine
21mConfiguration Languages Are the Bane of Our Existence