BI DataOps Development Process
summary
Describes a CSU DataOps pipeline that ingests PeopleSoft/oracle and CSV into parquet storage on S3, transforms with EMR + PySpark processing using a shared framework of launcher/config/common libs, and ships curated sets to Postgres for QuickSight. CI/CD uses CodeCommit, pull requests, CodeBuild tests and state machine scheduling; test automation uses PyTest + Kispa test harness comparing S3 sample inputs to expected outputs. Refactoring centralizes SQL/joins into lookup tables, reduces duplication, and yields ~30% functional code coverage.
AI-written summary · about
chapters (9)
- 0:00Introduction and overview
- 1:58High-level architecture layers
- 5:11Ingress data tools and formats
- 6:21Data processing framework
- 8:24CI/CD integration workflow
- 14:05Code refactoring benefits
- 18:19Test automation process
- 22:14Egress data to BI tools
- 25:53Q&A session
related talks
31mHow to Implement DataOps using
27mDataOps & Full Snowflake Automation In 1 Click
28mUsing Dataops To Build And Manage Data Products On Snowflake
33mAccidental DataOps: How we matured our organization with data, and how you can too!
30mData Engineering building your BI infrastructure from scratch
28mDevOps in a Data Warehouse: Inside Out
42mWeb Development To Big Data. A Journey
41mData DevOps is Real!