Can Data & ML really learn from software engineering?
summary
Argues that data and ML can borrow concrete software practices and also teach software about data as first‑class product, showing patterns like replacing ETL with ELT, defining data contracts as YAML schemas, and using dbt for SQL transformations with tests, lineage and branch comparisons. Gives examples—storing user search queries that improved product insight—and cites a survey that companies up to 1,000 people often have at most 10 people owning data, explaining why full SE practices carry real manpower costs.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:10Speaker background
- 3:22Data in every product
- 6:34Software vs data work
- 10:51Data for analytics challenges
- 13:28Engineers struggle with data
- 17:02Data contracts solution
- 19:32What engineers should learn
- 21:00Testing data pipelines
- 23:26Staging for data teams
- 25:54CI/CD for data training
- 26:58Modern data stacks
- 29:50Documentation & architecture
- 32:03Team size constraints
- 33:08Key takeaways
related talks
41mCan Data & ML really learn from software engineering?
27mA Look Inside Applied Machine Learning
27mData Creation: How Organizations Are Deliberately Creating Data To Power ML & Data Applications
21mHow The Application Of Machine Learning Is Changing The Way Companies Use Data
28mDataEngBytes 2023 - PER-T1-05 - Ellie Gholami
06mMachine learning is about your data and deployment
44mMachine Learning - Opening Pandora's Box
03mDeep learning at scale: A field manual