Schemas: The Key To Data Happiness
summary
Shows concrete tooling and outcomes: uses JSON Schema validate to catch missing required fields, plugs validation into an Airflow DAG to either fail or quarantine bad batches, and relies on Confluent Schema Registry for Kafka flow control. Demonstrates generating data with Faker and AI to generate four JSON records, auto‑generating SQLAlchemy and dbt boilerplate, and inferring schemas with PySpark and Jensen then diffing them (example mismatch: inferred long vs expected integer). Notes default JSON Schema allows extras unless additionalProperties: false, and recounts a real costly missed‑field bug from a Medicaid pipeline.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:39Who is Sev
- 6:22Validate
- 12:29Generate
- 12:40Faker
- 13:31Code Example
- 16:51Sample Data
- 17:19Unit Testing
- 18:39Generating Code
- 19:21dbt
- 20:12schema extraction
- 21:20detecting schema drift
- 23:24applying a schema vs validating data
- 25:32validating data with extra fields
- 26:32when to use
related talks
25mFull-stack forms with JSON Schemas
25mRecursion Schemes Without the Barbed Wire (Part 5)
04mLightning Talk JSON Schema
24mRecursion Schemes Without the Barbed Wire (Part 6)
30mContracts, Schema Evolution in Data Pipelines
21mRecursion Schemes Without the Barbed Wire (Part 4)
17mOpen Source for Data Quality
27mLive in the Data Wild West: The Data Contracts Sheriff