~/talk

Data Observability with HEDDA.IO

on YouTube 21m
Talkat SQLBits 2023 · 2023-03-14..03-18· [observability, data-engineering]

speakers

name:Oliver Engels
links:none on file
name:Tillmann Eitelberg
website:http://www.decompose.io

resources

video:https://youtu.be/RM4_BlfICPk
slides:none on file

summary

HEDDA.IO deploys into an Azure subscription and runs validation from Synapse/Databricks notebooks, producing per-run overall scores (example: Yellow Taxi demo shows 50% score) and processing ~20 million records per execution. It embeds pandas‑profiling for Spark to persist column profiles and correlations, supports row‑level business rules plus dataset rules (e.g., percent‑missing thresholds), exposes a REST API, alerts to Teams/Event Grid/DevOps, and provides a Web Runner that executes rules on parquet/CSV with actionable stats.

AI-written summary · about

chapters (7)

  1. 0:00Data observability with HEDDA.IO
  2. 3:39Data quality stats in HEDDA.IO
  3. 7:40Data profiling insights in HEDDA.IO
  4. 11:02Value insights and REST API
  5. 13:52Dataset rules for business context
  6. 14:56Alerting in HEDDA.IO
  7. 18:12DevOps integration and automation

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.