~/talk

DataEngBytes 2023 - PER-T1-01 - Chad Sanderson

on YouTube 30m
Talkat DataEngBytes Perth 2023 · on Tuesday, 2023-08-22· [data-engineering, machine-learning]

speaker

name:Chad Sanderson
links:none on file

resources

video:https://youtu.be/6fOLKX3XMk4
slides:none on file

summary

Claims that YAML contracts in Git plus CI/CD GitHub Actions enforcing tiers turn data quality from reactive toil into preventative control by validating schema, semantics and SLAs before deploy. Implementation at Convoy compares developer migration output to contract, tags downstream consumers in PRs, and integrates with Amundsen catalog and monitoring; tier 0 breaks the build, tier 1 alerts producer and consumer, tier 2 alerts consumer. Problem framing cites monthly outages costing millions and that >75% of teams rank data quality top investment.

AI-written summary · about

chapters (12)

  1. 0:00AI data quality crisis
  2. 2:07Poor data breaks AI
  3. 8:11Low data quality causes
  4. 11:21Why conventional fixes fail
  5. 13:48Preventative solution requirements
  6. 14:29What is a data contract
  7. 19:43Implementing contracts with YAML
  8. 21:30Enforcing contracts in CI/CD
  9. 24:42Where to enforce contracts
  10. 26:26Contracts with catalogs and monitoring
  11. 27:09Changing culture with contracts
  12. 29:17Key takeaways and Q&A

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.