Unit Testing Data with Marbles
summary
Argues that data needs executable assumptions and shows Two Sigma’s open‑source Marbles extending Python unit tests to capture local variables, add semantic assertions, and log every assertion as JSON (Elasticsearch‑friendly). Demonstrates with bike‑share CSVs and pandas, catching unexpected cases like trips longer than 10k or a marathon and month‑long rides. Provides notes on why a test exists, what to do, and time‑series of past failures so tests stay actionable; the system runs with existing unittest suites and CI.
AI-written summary · about
chapters (11)
- 0:00Data assumptions matter
- 4:33Capturing assumptions in tests
- 9:30Turning assumptions into tests
- 13:59Testing real bike share data
- 17:57Context loss in failures
- 23:14Marbles for richer failures
- 26:05Semantic assertions
- 30:10Preserving test context
- 34:18Marbles for unit tests
- 35:45Documentation and resources
- 37:21Q&A
related talks
40mUnit Testing Data with Marbles
01h 30mAdvanced Software Testing for Data Scientists
44mJust Another Marble Monday – Unit Testing NGRX/RxJS w/Marbles
01h 29mAdv. Software Testing for Data Scientists
25mTesting Data Pipelines
02h 46mThe Joy of Testing
36mHow Testability Drives Observability
46mUnit Testing by Example