Preflight Your Data: Catch Hidden Issues Before Handoff
summary
A practical preflight shows a small Python validator that runs a YAML rule pack against CSV/TSV sources, produces JSON and Markdown reports, and enforces severity levels so fails block submission while warnings can be acknowledged. Examples include duplicate occurrence IDs, invalid ISO dates, suspicious coordinates, and missing cross-file links. The author starts with 5–10 rules, demonstrates deterministic outputs and hashing so reruns show “resolved since last run,” and ships a human-readable report that points to exact rows to fix.
AI-written summary · about
chapters (10)
- 0:00Pre-flight data validation
- 1:11Hidden data issues survive
- 4:24Manual workflow inefficiencies
- 5:28What pre-flight validation means
- 7:38Starting with validation rules
- 9:29Python architecture for checks
- 12:04Human-readable reports
- 13:51Demo of ferry tool
- 15:42Benefits of reusable validation
- 16:47Q&A
related talks
35mBad data - anecdotes and examples from the real world
29mSchemas: The Key To Data Happiness
01h 34mKeep the Data Flowing
29mHigh Performance Data Loss
17mOpen Source for Data Quality
01h 29mImproving Open Data Quality using Python
27mTrallie: Shaping Unstructured Data Into Valuable Information
25mBOF: Vulnerability Data Consumers