~/talk

DataEngBytes 2023 PER-T1-10 - Matthew Carter

on YouTube 20m
Talkat DataEngBytes Perth 2023 · on Tuesday, 2023-08-22· [data-engineering, devops-and-delivery]

speaker

name:Matthew Carter
links:none on file

resources

video:https://youtu.be/d-xUoi66UEk
slides:none on file

summary

CICD pipelines run automated checks that catch surprises before users see them: merge to GitHub triggers GitHub Actions (or CircleCI/Codefresh) which deploys to a test environment, runs end-to-end tests with pytest and gherkin to catch logic flips (example: a transform suddenly reporting $1,000,000 profit vs negative cash), runs RBAC checks against Snowflake via parameterized pytest to block privilege escalations, and runs data quality checks with Soda or DBT contracts (types, nulls, freshness); alerting goes to Slack/Teams and tests must be designed to fail noisily and finish quickly.

AI-written summary · about

chapters (8)

  1. 0:00Introduction
  2. 1:03CI/CD explained
  3. 2:48End-to-end testing
  4. 8:06Security testing
  5. 11:36Data quality testing
  6. 14:46Why testing matters
  7. 16:53Alerting strategies
  8. 19:20Summary

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.