Data Pipelines for Operational Metrics
summary
OM1 builds an as‑code ELT pipeline that pulls monitoring, CI/CD, test, security, cost, incident and issue data into a Snowflake lakehouse using Airbyte with 350+ connectors and S3 ingestion, transforms raw loads with DBT for ELT transformations, orchestrates jobs via Airflow, and links resources by tags and Okta group snapshots so costs map to repos and teams. Outputs include budget “worm” charts, DORA metrics, executive dashboards, and a GitHub‑action lambda that enforces GitHub PRs must reference Jira via lambda; the system reduces manual spreadsheets and even eliminates engineering timesheets.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:57Key data concepts
- 6:30Identifying data sources
- 11:39Building pipeline workflow
- 16:53Extracting raw data
- 20:45Transforming with DBT
- 26:38Reports and dashboards
- 31:30DORA metrics applications
- 32:54Lessons learned
- 34:43Q&A
related talks
21mUsing DBT and Python to Build Production Pipelines in Snowflake
29mData Acquisition: Overcoming Challenges
31mHow to Implement DataOps using
42mThreat Detection across all environments with SnowflakeData Security Lake
28mUsing Dataops To Build And Manage Data Products On Snowflake
20mBuilding a Data Pipeline with Testing in Mind
33mAutomating Data Analytics: Collect, Transform and Deploy - Lead With Change
28mAutomating Database Operations with Cortex Code