Scale-up your job satisfaction, not your software
summary
Shows how to turn vague data projects into usable products by defining business requirements with users, adding schema validation at ingestion, and running fast unit and integration tests instead of relying only on slow end-to-end suites. Describes building CI pipelines that enforce model versioning and startup checks so bad inputs or missing startup.sh never reach production, with audit logs and rollback paths. The result is fewer wasted builds and deploys and reproducible, trustworthy data outputs.
AI-written summary · about
chapters (6)
- 0:00Introduction and motivation
- 2:58Testing culture shift
- 7:53Performance vs correctness
- 13:45Version control and testing
- 17:52Automation and feedback loops
- 19:42Final thoughts and resources
related talks
18mData quality: the key to long term happiness
38mKeeping pace with data infrastructure so it doesn't outrun you
31mWhen the data levee breaks: Managing cognitive load in data-intensive projects
07mFrom Scaling APIs to Scaling Data to Scaling Your Team
20mShift-left testing : Building reliable Data Pipelines
29mManaging your Data Assets: The interview
24mDistinguishing features of production quality pipelines
42mBringing Software Engineering Rigour to Data, DataEngBytes 2022