Keeping pace with data infrastructure so it doesn't outrun you
summary
Advocates embedding a software development lifecycle into data work, treating models as code with versioning, unit tests for table creation, and automated monitoring that triggers retraining; cites Fast.co running 500,000 lines of code, 75+ engineers and 25+ pipelines to justify processes. Recommends tools like dbt, Spark, Snowflake, DynamoDB, GitHub Actions and CI to separate infrastructure from business logic and enable auto-generated documentation and self-serve analytics.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:05Speaker background
- 3:09Evolution of data infrastructure
- 6:00Software development lifecycle in data
- 9:56Models as versioned artifacts
- 11:01Monitoring model degradation
- 12:04Documentation for data democratization
- 15:46Investing in data tools
- 19:21Closing thoughts
- 22:09Q&A session
related talks
42mBringing Software Engineering Rigour to Data, DataEngBytes 2022
40mData Engineering in a Rapidly Changing World
38mData Secrets from a Platform Engineer
57mBACK TO THE FUTURE: EMERGING TRENDS IN DATA ENGINEERING
38mThe Rise & Downfall of the Data Engineer REVISITED
34mTales of Data Architecture Evolution
39mHelp Me, Help You: Supporting Your Data
27mData team strategies: unleashing your team